Compare commits

...
Author SHA1 Message Date
Chris LuandClaude Opus 5.5 b89d7e36bf install.sh: offer the vector engine on linux only
Releases build weed-vector for linux amd64/arm64 only, so --component vector
on darwin or windows downloaded a missing asset.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
2026-11-14 11:26:49 -08:00
b067266e6b shell: fix s3.circuitBreaker help examples and error messages (#12657)
* shell: fix s3.circuitBreaker help examples and error messages

The help examples used -type count, but -type accepts Count and MB (as
its flag description says), so both examples failed with "type not
valid, only 'count' and 'bytes' are allowed", an error that names
neither accepted value. Two other errors pointed at flags that do not
exist: -max and -counts, where the flag is -values.

Use -type Count in the examples, name Count and MB in the type error,
and -values in the value errors. TestCircuitBreakerHelpExamplesRun runs
every help example and fails on master for the two -type count lines.

Signed-off-by: anant <170962632+antcybersec@users.noreply.github.com>

* shell: name the bad action and the -actions flag in s3.circuitBreaker

The unknown-action error pointed at a nonexistent -action flag and echoed
the whole -actions list instead of the action that was rejected.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Signed-off-by: anant <170962632+antcybersec@users.noreply.github.com>
Co-authored-by: anant <170962632+antcybersec@users.noreply.github.com>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-14 11:17:52 -08:00
b3a58506aa filer: report a SQL listing that fails partway through (#12654)
* filer: report a SQL listing that fails partway through

ListDirectoryPrefixedEntries never checked rows.Err() after its rows.Next()
loop. When the result set failed partway, for example on a lost
connection, rows.Next() returned false and the listing returned the
entries read so far with a nil error, so callers saw a shorter directory
instead of an error. Check rows.Err() once the loop has run to the end,
as the scan in abstract_sql_store_scan.go already does. A listing the
callback stopped on purpose is left alone: rows.Err() would return a
canceled request context's error there.

This covers every store built on AbstractSqlStore (mysql, mysql2,
postgres, postgres2, sqlite).

Signed-off-by: anant <170962632+antcybersec@users.noreply.github.com>

* filer: wait for the canceled SQL listing to release its rows in the test

A fixed sleep let the callback return before database/sql closed the
rows, so a listing that always consulted rows.Err could still pass.
Wait, bounded, until the connection is back in the pool instead.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Signed-off-by: anant <170962632+antcybersec@users.noreply.github.com>
Co-authored-by: anant <170962632+antcybersec@users.noreply.github.com>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-14 11:14:54 -08:00
anantandanant b42993d24a shell: make TestCircuitBreakerShell compare against the expected configs (#12656)
The test unmarshalled the command's output into both actual and expect,
so it compared the output with itself and the expected JSON in each case
was never checked; it also skipped the first case. Compare every case's
output with its expected config as S3CircuitBreakerConfig messages (the
output includes default fields such as "enabled": false that the
expected JSON leaves out), and restore LoadConfig when the test ends.

Signed-off-by: anant <170962632+antcybersec@users.noreply.github.com>
Co-authored-by: anant <170962632+antcybersec@users.noreply.github.com>
2026-11-14 11:14:14 -08:00
Chris LuandClaude Opus 5.5 fd4603d657 s3 lifecycle: count filer entries a walk reads but skips as progress (#12658)
* dailyrun: count entries read from the filer, skipped ones included

A versioned bucket's pre-pass and a resumed walk read entries without
emitting them, so a walk can read for hours with no visited entry.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3 lifecycle: count listed entries as progress work

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-14 11:12:33 -08:00
Chris LuandClaude Opus 5.5 53377bcb5d mount: keep a handle dirty when a write lands during its flush (#12652)
A flush uploads the dirty pages without the handle lock, then commits the
entry under it. A write in between grows the entry's size, but its pages
are not in the committed chunks. Clearing dirtyMetadata then made the next
flush skip the commit, leaving a file with the new size and no chunk for
the write, which reads back as zeros.

Count page writes on the handle and clear dirtyMetadata only when none
landed since the flush started uploading.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-14 11:11:01 -08:00
1167ca8cf5 vacuum: report a request dropped because another vacuum is running (#12648)
* vacuum: report a request dropped because another vacuum is running

The master runs one vacuum at a time. A request that arrives while
another one is running, the automatic vacuum or an earlier request, is
dropped: Topology.Vacuum logs "Vacuum is already running" and returns.
The VacuumVolume RPC and /vol/vacuum then answered as if the vacuum had
run, so volume.vacuum printed nothing and exited 0 with no volume
compacted. Where the automatic vacuum is busy for hours, an explicit
volume.vacuum -volumeId could do nothing every time, and nothing said so.

Topology.Vacuum now returns ErrVacuumRunning for a dropped request. The
RPC returns it and /vol/vacuum answers 409 with it. volume.vacuum names
the request that failed, because a list of volumes or collections is
sent as several requests and a vacuum that starts in between fails only
the later ones. Requests are still dropped, not queued, and the
automatic vacuum ignores the error as before.

The integration tests that need their /vol/vacuum request to run ask
again while the master answers 409.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>

* docs: note the 409 answer from /vol/vacuum while another vacuum runs

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
2026-11-14 11:07:44 -08:00
Chris LuandClaude Opus 5.5 512f898600 s3 lifecycle: report work-based job progress during a pass (#12644)
* s3 lifecycle: report work-based job progress during a pass

The daily pass sent progress only while starting, so a healthy scan that
ran for many hours could reach the admin's 24-hour no-progress guard.
Worker heartbeats prove the worker is alive, not that this job moves.

While dailyrun.Run is in progress the handler now sends a progress update
every 5 minutes, and only when the pass's counters (entries visited,
replay events, RPC outcomes including errors) moved since the last
delivered update. An idle tick sends nothing, so a stalled pass still
expires. A failed send is retried on the next tick. The reporter is
stopped and joined before the completion is sent, so no progress can
follow the terminal result.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* admin: test that progress, not heartbeats, keeps an old job active

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3 lifecycle: label progress phase and bucket as the last started

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-14 10:55:08 -08:00
Chris LuandClaude Opus 5.5 858f8cbad8 s3 lifecycle: check internal S3 authentication before starting a pass (#12645)
* s3 lifecycle: add a LifecycleReady RPC that only authorizes the caller

LifecycleReady runs the same checkAdminAuth as LifecycleDelete and reads
no bucket or object metadata, so a lifecycle worker can confirm that an S3
front accepts its internal credentials before it starts a pass.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3 lifecycle: check S3 fronts with LifecycleReady before the pass

Execute used to dial the first registered S3 front and start loading
policies and walking buckets, so a missing or mismatched
jwt.filer_signing.key surfaced only as repeated per-bucket delete
failures. It now checks each front in order with LifecycleReady, using
the credentials LifecycleDelete sends, under a 10s bound that also
follows parent cancellation. The first front that passes serves the
pass; fronts that fail are reported in the job's progress and the log.
When none passes, the job fails before any listing with an error that
names the front and says whether this worker has the key set, without
key or token material. Fronts that predate LifecycleReady are checked
with an empty LifecycleDelete, which they authorize and then reject
before any lookup.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-14 10:53:07 -08:00
Chris LuandClaude Opus 5.5 c819e3fbea s3 lifecycle: per-pass run results and last full-pass status (#12643)
* s3 lifecycle: collect per-pass bucket results and work counters

WalkBuckets kept only the first bucket error and logged the rest. A
RunStats shared by the walker, the walker dispatcher and the replay
drain now records every bucket's walk result (walked, failed, skipped
on cancellation), entry and object visits, replay events, and
LifecycleDelete outcomes under the dispatch_total outcome labels.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3 lifecycle: persist the last fully successful pass

Shard cursors and walk timestamps advance on partial passes as well, so
neither says when lifecycle last covered every policy. Keep that in its
own record next to the cursors, plus a worker gauge for it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3 lifecycle: report each pass's result in the job record

The job ended with a fixed success summary, or with only the first error,
and malformed lifecycle policies were V(1) logs. The completion now
carries the pass status (complete, partial, failed, canceled), bucket
totals and failures, invalid policies, entry, object and event counts,
LifecycleDelete outcomes, the last phase and bucket, the elapsed time
and the last full pass, on failed passes as well.

Only a complete pass advances the full-pass record; a malformed policy
makes the pass partial.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* admin: keep a failed job's result in its record

Completions with Success=false dropped Result.Summary and
Result.OutputValues, so a handler could not report what a failed run
had done, which buckets failed, or how far it got. Keep them; the error
message still drives the job's message and run record.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* grafana: chart the time since the last full lifecycle pass

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3 lifecycle: a replay halted on an unresolved delete is a partial pass

BLOCKED, RETRY_LATER and exhausted transport retries stop a shard's
replay without an error, leaving its due events for a later pass. Count
those shards and report the pass as partial, so it no longer advances
the full-pass record.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3 lifecycle: a pass with throttled walks is not a full pass

WalkerInterval can skip the walk for walk-only rules (ExpirationDate,
ExpiredDeleteMarker, NewerNoncurrent) with no error. Count those shards
and keep the full-pass record for passes that evaluated every rule; the
job reports full_pass and walks_deferred.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3 lifecycle: keep the saved full-pass time when the save fails

Publishing a timestamp that never reached the filer let the gauge move
back on the next job's reload. Keep the previous value and report the
save error in the job result.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* admin: drop a failed attempt's summary when the retry reports none

Retries reuse the job record, and failed completions now keep their
result summary, so a retry that completes without a Result showed the
failed attempt's summary, and a successful one used it as its message.
Clear the summary when a completion carries no Result, as the output
values already are.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-14 10:50:37 -08:00
Chris LuandClaude Opus 5.5 6ad4b07598 mount: scope and order the final-release writeback fence (#12651)
* mount: fence a dropped handle only when it holds unpublished content

dirtyMetadata is also set by chmod, xattr and fallocate, so a failed
metadata-only close refused every later write of the inode.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* mount: fence the inode before its last handle leaves the handle map

A reopen between the map removal and the fence could publish at the
stale kernel EOF.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* mount: fence a failed metadata flush only when the handle holds content

A refused synchronous flush or an abandoned writeback metadata flush of a
handle with only chmod, xattr or fallocate changes fenced the inode, so
later writes were refused until FORGET.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* mount: count a size change on an open handle as unpublished content

A truncate or extend through an open handle changes the entry's size and
chunks, but it set only dirtyMetadata. A failed close then skipped the
final-release fence, so a reopen before FORGET could publish against the
untruncated entry.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* mount: mark copied content dirty before each page of a streaming copy

A copy that failed or was cancelled after earlier pages were accepted left
dirtyContent false, so the final-release fence skipped that content.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* mount: mark copied content dirty only after a page is accepted

A copy whose first page is rejected leaves the destination unchanged and
the kernel EOF where it was, so fencing the inode on release only blocked
later writes until FORGET.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* mount: skip the server-side copy while the destination holds unpublished content

A whole-file server copy clears dirtyContent when it commits, so a page a
cancelled streaming copy left behind would lose its release fence. Fall
back to the streaming copy instead.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* mount: skip the server-side copy while the source holds unpublished content

The filer copies the stored source, so pages still pending on the source
handle would be missing from the destination. The streaming copy reads
them through the dirty pages.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* mount: check the publication fence before a server-side whole-file copy

The filer copy publishes the destination without going through the
flush-time guard, so a fenced writeback inode could still take new
content before FORGET. Fall back to the streaming copy, whose flush is
guarded.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-14 10:47:33 -08:00
Ping Qiu 65ecb6aa12 fix(mount): prevent negative activity-score shifts (#12631)
fix(mount): bound activity decay shifts across time regressions
2026-11-14 10:46:57 -08:00
Chris LuandClaude Opus 5.5 0a38738076 filer: keep the listing snapshot below later and unfinished event stamps (#12642)
filer: take the listing snapshot as a claim below later and unfinished stamps

A ListEntries snapshot tells the client that every event at or below it is
already in the listing, so the mount drops those events from a directory
build and fences them with the directory floor. The snapshot was a raw
time.Now(), which an event stamp can tie on a coarse clock: on Windows a
mkdir right after a listing of its parent was stamped with the snapshot's
own timestamp, the mount took it as listed, and the new directory stayed
missing from a parent marked cached. A create stamped before the snapshot
but not yet in the store was dropped the same way.

Take the snapshot through the same in-flight claim the idle heartbeat uses:
it stays below every unfinished stamp, and every later stamp lands above it.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-14 01:17:51 -08:00
10f8b2d5b1 mount: guard peer-truncate gaps, stale append EOF, and terminal writeback refusal (#12630)
* mount: guard append gaps over concurrent content merges

* mount: fence a writeback inode when its async publication is abandoned

A drain that gives up on a non-precondition error left the token clear, so a
reopened writeback handle could publish past the stored EOF.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* mount: fence a writeback inode when its async data flush fails

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* mount: fence the inode when its last handle drops uncommitted content

A synchronous release whose final flush fails destroys the handle's dirty
pages, while under writeback the kernel EOF already counts them.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-14 01:13:37 -08:00
b50d4a9cab mount: don't spend the close()/fsync commit deadline on uploads (#12641)
* mount: start the close()/fsync commit deadline after the data upload

Flush, Fsync and the truncate path of SetAttr create the 30 s
metadataFlushTimeout context on entry and pass it to doFlush, so the
clock also runs through FlushData, the upload of the dirty pages. That
upload has its own timeouts and retries. When an assign through a stalled
filer or a slow volume server takes most of the 30 s, CreateEntry then
fails at once with "context deadline exceeded" and close() returns EIO
for data that was already stored. The deadline is meant for the commit
alone (#10102: "a healthy CreateEntry completes in well under a second").

doFlush now takes the commit timeout and starts the deadline once the
data is uploaded. close() still cannot hang on a stuck filer: the commit
keeps its 30 s bound. Release and the flush before a rename pass 0, as
before (no overall bound, only the per-call filer timeouts).

Seen in production on a training cluster: a 30 s stall of one replica's
disk, and a filer that stalled for about 30 s, each failed close() with
EIO although the upload had succeeded.

Tests: close(), fsync() and a truncate below dirty data, each after an
assign slower than the deadline (all three fail before this change), and
a filer that never answers CreateEntry still fails close() at the
deadline.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* mount: charge the flush commit deadline only for its filer calls

Starting the 30 s deadline after the data upload is not enough.
flushMetadataToFiler does more than call the filer before it commits: it
waits for the handle's locks, reads the file back and uploads it again
when its chunks are bloated (maybeMergeChunks), and uploads a manifest
once enough chunks have built up (MaybeManifestize). None of that takes
the context. When it outlasts the deadline, CreateEntry again fails at
once with "context deadline exceeded", and close() or fsync() returns an
error for data that was stored.

The deadline becomes a budget, commitBudget, charged only while a filer
call of the commit runs: CreateEntry, and the read of a changed hard
link before it. The attempts of one flush share it, and a flush is not
retried once it is spent, so a stuck filer still fails close() after
30 s. Release, the flush before a rename and the async flush pass no
budget, as before.

Seen in production: the merge of a 1.8 MiB sqlite file with 183 chunks
took 32 s because one chunk's read first tried a volume server that was
down, and the commit failed 32 microseconds after the merge finished.

Tests: a flush whose manifest upload is slower than the deadline commits
(it fails without this change), and the filer calls of one flush share
the budget.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>

* mount: give the stalled-lookup refusal test the new doFlush argument

TestContentFlushRefusalWithStalledLookup came in from master after this
branch changed doFlush's signature, so the merged package did not build.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
2026-11-13 21:56:44 -08:00
Chris LuandClaude Opus 5.5 e0890391ac mount: keep a refused content create refused when the stored-entry read stalls (#12629)
The read of the stored entry ran in the create's filer callback, so a read
that hit the call deadline turned the refusal into a filer timeout, and
both retry layers resent the refused create. Read it in its own call,
bounded to one filer call.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 20:19:35 -08:00
Chris LuandClaude Opus 5.5 6eb2fb4842 filer: stop full-chunk reads at the chunk size after a mount truncate (#12632)
* util/http: stop a full-chunk read at the chunk's recorded size

weed mount shrinks a chunk's Size in place on ftruncate but leaves its
needle untouched. A view covering the whole shortened chunk is fetched
without a Range header, so the reader returned every stored byte. The
filer GET then wrote past the declared Content-Length and sent an empty
body, and the stream reader served the stale tail.

ReadUrlAsStream and the cipher path now deliver at most size bytes;
size 0 still reads the whole needle for manifest and log chunks.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3api: limit a plain chunk read to the requested size

A full-chunk read sends no Range header, so a chunk shortened by a mount
truncate streamed its whole needle into the S3 response.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* util/http: still report a failure past the chunk size

The volume server aborts a full-needle response when the needle fails
its checksum, and only at the end. Stopping at the chunk size missed
that, so a damaged copy read as healthy. Read the rest of the body and
return its error so the caller tries another copy.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 20:17:46 -08:00
Chris LuandClaude Opus 5.5 63c99183d1 s3vectors: name weed-vector processes servers (#12627)
* s3vectors: call the registered weed-vector processes servers

The gateway's discovery named them workers, the name of the admin
plugin workers. Also drops two comments left without a function.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* test/s3vectors: build the Enterprise engine as weed-vector

In an Enterprise checkout, make ./weed-vector builds the distributed
engine, which also serves VectorService, instead of a separate
weed-vector-worker. An open-source checkout builds weed-vector-server
as before.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* test/s3vectors: distributed mode starts two more weed-vector engines

It started weed-vector-worker, which no target builds now, and waited
for a gateway log line that no longer exists. The Enterprise
weed-vector's engines all join the placement controller.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 20:17:15 -08:00
Chris LuandClaude Opus 5.5 a8436a621a admin: vector servers on the dashboard (#12606)
* admin: vector servers on the dashboard

A Vector servers count in the Cluster card and a Vector Servers table
after S3 Servers, from the engines the dashboard already discovers.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* admin: drop the dashboard's vector server kind

Every vector server is a weed-vector engine, so the column said the
same thing on every row.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 20:16:43 -08:00
Chris LuandClaude Opus 5.5 58d25998d0 admin: Cluster → Vector Servers page (#12603)
Lists the weed-vector engines registered with the master, with version
and health, next to the other server lists.


Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 20:16:00 -08:00
github-actions[bot] d91d58ce47 docs: regenerate star history chart 2026-11-14 02:04:51 +00:00
Chris LuandClaude Opus 5.5 4fc41872e3 mount: keep an open file publishing when a peer unlinks one of its hard link names (#12613)
* mount: record a hard link's looked-up names on its inode

Lookup mapped a second name of a hard-linked file to the existing inode
without adding it to the inode's names, so GetAllPaths missed it and the
final forget left it mapped to a released inode.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* mount: keep an open file publishing when a peer unlinks one of its names

A delete event from another client marked the inode's open handle deleted
even when other names still held the hard-linked file, so every later
fsync reported success without saving metadata. The delete event's entry
carries the link count the filer decided on, so an alias delete now drops
only that name, keeps the handle on a name this mount knows, and has the
next flush read the survivor's stored entry before publishing.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* mount: test peer unlinks of a hard link name held open

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* mount: leave an unlinked file out of the periodic metadata flush

flushFileMetadata did not check the handle's delete mark. A file
unlinked while open before its deferred create reached the filer was
created again by the next periodic flush, which expects no entry.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* mount: read a pending link change before a periodic metadata flush

Only flushMetadataToFiler applied linkRefresh. After a peer unlinked one
name, the periodic flush kept publishing over the old link count, so the
filer refused every one until a regular flush ran.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* mount: test the periodic flush after a peer unlinks a hard link name

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 16:14:22 -08:00
Chris LuandClaude Opus 5.5 08f9dcb6de admin: vector buckets link maintenance jobs to the Lance lane (#12633)
The Vector Engines and Maintenance card linked to the general plugin
monitoring page; the vector index jobs run in the Lance lane.


Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 16:14:19 -08:00
Chris LuandClaude Opus 5.5 8a756af348 vector: key shared leases by manifest commit time too (#12626)
On Windows Lance commits local tables by rename and leaves the manifest
ETag empty, so a table dropped and written again at its location reused
the old table's lease for the same version number.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 15:52:52 -08:00
Chris LuandClaude Opus 5.5 68b6a2ed08 mount: model the refused async flush as a replaced file in recovery tests (#12628)
A peer's chunk append now merges into the flush, so these tests no longer
saw the refusal they exercise. A replaced file is still refused.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 13:50:51 -08:00
anantandanant fdd3a692d3 shell: show -collection, not -c, in ec.balance and ec.rebuild help (#12617)
Both usage lines showed [-c EACH_COLLECTION|<collection_name>], but the
flag is -collection; there is no -c. ec.rebuild's own Options section
already lists -collection.

Signed-off-by: anant <170962632+antcybersec@users.noreply.github.com>
Co-authored-by: anant <170962632+antcybersec@users.noreply.github.com>
2026-11-13 13:03:07 -08:00
Chris LuandClaude Opus 5.5 45cd6033a0 s3: make conditional DeleteObject atomic on versioned buckets (#12624)
* s3: re-check a versioned DeleteObject If-Match under the object lock

A versioned or suspended DeleteObject without a versionId routed to the
owner filer after checking If-Match outside any lock, so concurrent
conditional deletes could all pass the check and each add a delete
marker. Keep conditional deletes on the lock path, which re-checks the
precondition under the object write lock, as conditional versioned PUTs
already do.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3: guard a conditional version delete on the owner filer

DeleteObject with a versionId checked If-Match before routing the
delete to the owner filer, so concurrent conditional deletes of one
version could all report success. Carry the If-Match clause into the
owner's transaction as a guard on the version (the null entry for
versionId=null), evaluated under the object's path lock before anything
applies, and map that guard failing to 412 instead of the object-lock
403. An If-Match the filer cannot evaluate stays on the lock path.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3: carry an If-Match ETag list into the routed delete

A DeleteObject whose If-Match listed several ETags fell back to the
gateway's object write lock, while a single-ETag delete of the same key
ran under the owner filer's path lock. The two locks do not exclude each
other, so a list delete and a single-ETag delete could both report
success. Send every candidate to the filer, whose strong comparison
matches checkDeleteIfMatch, so all conditional deletes with an owner are
decided under the owner's lock.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3: fail a conditional null delete on an owner transport error

The owner may have committed the delete before the error, and the gateway
lock path it fell back to does not exclude routed deletes, so a second
If-Match delete could pass there too. Answer 500 as the specific-version
path does; unconditional deletes still fall back.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3: fail a conditional null delete on any owner error but the nested refusal

Only the owner's non-empty-folder refusal is known to apply nothing. After
any other owner error the delete may have been partly applied, and a lock-path
re-check does not exclude a concurrent routed delete of the same version, so a
conditional null-version delete answers 500 instead of falling back.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3: strip a nested conditional null delete on the owner

A conditional versionId=null delete that the owner refuses because other
keys are nested under the entry used to fall back to the gateway lock and
re-check If-Match there. That lock does not exclude routed deletes: once
the last nested key is removed, a concurrent routed delete can pass the
same check on the owner, and the fallback then treats the missing entry as
deleted, so both requests answer 204.

Demote the entry on the owner instead, in a transaction guarded by the
If-Match clauses and by the entry it was read as, with the object-lock
guards on the request. A missing or already demoted entry answers 412.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* filer: add an IF_OBJECT_EXISTS write condition

IF_EXISTS also holds for a plain directory, which is what is left of a key
once the object on it is stripped while other keys stay nested under it.
IF_OBJECT_EXISTS holds only for an entry that carries an object.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3: require an object for a conditional null delete with If-Match: *

Stripping the null version off an entry other keys are nested under leaves a
plain directory. Once the last nested key is gone, a routed delete with
If-Match: * still found an entry there and removed the empty directory, so a
second conditional delete could also answer 204. The owner now checks that
the entry carries an object.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3: settle a wildcard null delete on the entry when the filer predates IF_OBJECT_EXISTS

A filer older than IF_OBJECT_EXISTS fails it closed, so with the gateway
upgraded first every If-Match: * delete of a null version answered 412. When
the narrowed guard fails, the gateway now reads the entry on the owner: absent
or a plain directory is a 412, otherwise the file is deleted, or the directory
stripped, under IF_EXISTS and IF_ENTRY_EQUAL on the entry it read. The demote
keeps the caller's If-Match clauses, since the entry guard already requires an
object.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 12:47:22 -08:00
d695d101b5 Fix/s3 abandoned chunk download (#12616)
* filer: cancel a chunk download that no reader waits for

* filer: stop waiting for reader cache budget once cancelled

* filer: keep a cancelled download's re-lookups off its context

The refresh and retry lookups took the download's context. FilerClient
coalesces concurrent lookups of a volume, so a cancelled download leading
one failed the other readers waiting on it. Fetches still stop on cancel.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* filer: test that cancelling a download leaves its re-lookup running

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* filer: cancel unheld downloads when the reader cache is destroyed

ReaderCache.destroy waited for every downloader, so closing a reader
waited for a stalled prefetch nobody read. Downloads with no reader or
stream on them are now cancelled first; ones still in use keep running.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* filer: test that destroying the reader cache cancels an unread prefetch

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* filer: cancel a destroyed download once its last holder leaves

destroy skipped downloads a reader or stream still held, and once they
were out of the map nothing cancelled them, so destroy waited for a fetch
nobody used. removeConsumedLocked now cancels any consumable download,
whether or not it is still in the map.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* filer: test a destroyed download cancelled after its last stream leaves

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* filer: release the stream only after destroy dropped the download

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* filer: test a cancelled stream leaving a waiting reader's download running

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 12:45:34 -08:00
Chris LuandClaude Opus 5.5 813ac74d79 s3tables: keep legacy-subject owners and policy principals for federated STS sessions (#12615)
* s3tables: re-anchor legacy owners for federated STS sessions

A web-identity role session was keyed on its bare OIDC subject until
owners became issuer-scoped. Its sessions now carry the scoped aws:userid
but no asserted issuers, so the legacy subject check never applied and the
session lost the table buckets and namespaces it had created. Read the
configured issuer set for such sessions.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3api: keep token claims from asserting trusted issuers on STS sessions

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* iceberg: give the catalog's table handler the local accounts and issuer set

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3tables: test a federated session on a bucket its legacy subject owns

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* sts: drop a token-supplied aws:userid when a session has no parent user

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3api: take a session's aws:userid from its signed parent user

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3api: test that session claims keep only the server-asserted aws:userid

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3tables: match legacy subject patterns in table bucket policy principals

A policy written before issuer scoping can name role sessions by a pattern
over their bare OIDC subjects. Match such a pattern against the caller's
subject claim once that subject resolves to the session's scoped principal
under the single configured issuer.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3tables: test a federated session against a legacy subject pattern principal

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 12:43:52 -08:00
Minh Vu 4bc7598a43 admin: escape dynamic text in S3 bucket modals (#12625)
admin: escape dynamic text in bucket modals
2026-11-13 12:42:40 -08:00
Chris LuandClaude Opus 5.5 d685709832 s3api: refuse writes to a stored version's physical path (#12622)
* s3api: refuse a PUT to a stored version's physical path

In a Suspended bucket a PUT to <key>.versions/v_<id> replaced the stored
version's body in place. Look the entry up when the key has the version
file shape and deny the write when it is a version file.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3api: refuse copy, multipart, POST and rename writes to a version file

Apply the same version file check to the CopyObject, CreateMultipartUpload,
CompleteMultipartUpload, POST Object and RenameObject destinations.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3api: read a version file write target on its logical owner

The guard resolved the key as an object, which on a ring reads the
physical path's owner and, in a versioned bucket, may return a nested
version under <key>.versions/v_<id>.versions instead of the stored
version. Look up the physical entry directly, routed by the logical key.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3api: confirm a version file miss on a skipped owner

When the owner was recently unreachable the routed lookup answers from a
replica, whose miss does not show the version is absent. Ask the owner
directly in that case and refuse the write when it cannot answer.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3api: confirm every routed version file miss on the owner

Whether the routed lookup skipped the owner can change between the lookup
and a second unreachable check, letting a replica's miss through. Ask the
owner after any miss instead.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 12:39:40 -08:00
d1a97f4c3d fix(mount): retain refused async metadata before releasing handle (#12598)
* fix(mount): retain refused async metadata before releasing handle

* mount: sync the failed-metadata directories with util.FsyncDir

Directory fsync fails on Windows, where the mount also builds, and returns
EINVAL on some filesystems. Either kept the retention loop retrying a record
that could never be saved, so the drain never finished.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* mount: keep failed-metadata records in a per-user directory

The write cache root defaults to the shared temp directory. The first mount
to save a record created failed-metadata there as 0700, so another user's
mount could never save one and its drain retried forever.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* mount: drop the async conflict probe test

TestContentAsyncConflictKeepsRecoveryRecord drives the same refused drain
with the retry backoff stubbed out; the probe slept through the real backoff
(7 s) and its kept-handle branch could not run. Its peer-entry check moves
into the recovery record test.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 12:34:58 -08:00
5a54009439 filer/etcd: client keepalive and per-request deadlines (#12621)
* filer/etcd: client keepalive and per-request deadlines

An etcd member that goes silent with its connections open (a dead or hung
node, a network partition, a frozen process) keeps the store's requests
balanced onto it waiting forever, because the client has no keepalive and
the calls have no deadline. Writes through the filer stop although the
other members keep quorum.

keepalive_timeout (default 3s, "0s" turns keepalive off) enables gRPC
keepalive with the ping interval fixed at gRPC's 10s minimum. gRPC also
sets TCP_USER_TIMEOUT to it on Linux, so a connection whose sent data goes
unacknowledged that long is closed by the kernel. request_timeout (default
3s) bounds every Get, Put and Delete unless the caller set a deadline;
timeout keeps its meaning as the dial timeout. DeleteFolderChildren gets no
request deadline: its DeleteRange grows with the directory, and keepalive
still ends it if the member goes silent.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Update weed/filer/etcd/etcd_store.go

Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>

* Update weed/filer/etcd/etcd_store_test.go

Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>

* Update weed/command/scaffold/filer.toml

Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>

* filer/etcd: no request deadline on entry writes

A Put whose client deadline expires may still have been committed. The
filer's upload handler treats the error as a failed write and deletes the
uploaded chunks, which the saved entry may then point at. InsertEntry and
UpdateEntry therefore keep the caller's context: etcd's own request
timeout ends a Put a live member cannot commit, and keepalive ends one on
a member that goes silent. Reads and deletes keep request_timeout; a
timed-out store delete returns before the filer deletes any chunks.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* filer/etcd: no request deadline on KvPut

The filer wrapper stores a hard-linked file's attributes and chunk list
through KvPut. An HTTP append to a hard-linked file whose KvPut times out
after etcd saved it returns an error, and the upload handler deletes the
appended chunks the saved hard link now points at. KvPut keeps the
caller's context like InsertEntry and UpdateEntry.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
2026-11-13 12:28:48 -08:00
Yip LeoandClaude Opus 5.5 911bfe4014 mount: stop overwriting the mount root's filer entry (#12623)
When a mount is rooted at an existing directory, two code paths replace
that directory's filer entry with one the mount builds itself:

- every mount start re-creates the root with a non-exclusive mkdir,
  which the filer handles as an update of the whole entry;
- creating or removing an entry directly in the root saves the root's
  entry to touch its mtime, and maybeLoadEntry synthesizes that entry
  from the mount options instead of reading the filer.

The filer keeps only crtime and inode on such an update, so the root
loses its mode, owner and extended attributes. When the root is a
bucket it also loses the bucket quota, which silently drops to 0
(unlimited), and the settings S3 keeps on the bucket entry. The explicit
allowEmptyFolders=false that #11370 meant to keep is replaced by true at
every mount start, because the mkdir has wiped it before
ensureBucketAllowEmptyFolders looks.

Create the root only when a lookup says it is missing, and then
exclusively. Leave the root out of the filer-side mtime touch: its
attributes come from the mount options (setRootAttr), so the filer copy
of its mtime was never visible through the mount anyway.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 12:27:15 -08:00
Chris LuandClaude Opus 5.5 737ef68a11 mount: keep publishing an open file through its surviving hard link (#12609)
* mount: keep flushing an open file whose other hard link survives an unlink

Unlink marks the inode's open handle deleted so no flush recreates the
removed name. When the entry was one of several hard links the inode lives
on under another name, yet the mark stayed and every later fsync and close
returned OK without publishing metadata. Lift the unlink's own mark once
the name is gone and the inode still has a path.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* mount: carry a link's stored entry into the file's open handle

Link rewrites the source entry with the new hard link id, count and times,
but an open handle kept its old base and entry. Its next conditional
content flush was refused (EIO), and an unconditional one would have
dropped the hard link id. Move the handle onto the stored entry when only
the link fields and times changed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* mount: carry an alias unlink's stored entry into the open handle

Removing one of several hard links lowers the stored link count and
stamps a new ctime, and the handle may still hold the removed name. Look up
the entry at the name the handle flushes through and adopt it, so the next
content flush matches the stored entry, writes the right count, and
attribute saves address the surviving name.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* mount: test an open handle across link and alias unlink

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* mount: read an alias unlink's lost entry again at the next flush

When the read of the surviving entry right after an alias unlink fails,
the handle kept a base with the old link count and ctime, and every later
flush was refused even after the filer recovered. Leave the hard link id
on the handle and have the next flush read the stored entry and adopt it
before publishing.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* mount: publish a queued close before unlinking a name another link keeps

With writeback caching, close only queues the flush. Unlink marked the
handle deleted before waiting for that flush, so the flush skipped its
metadata and the surviving link kept the content from before the close.
When another link keeps the file, wait for the queued flush first.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* mount: let a queued close reach the filer before linking the file

With writeback caching a new file is created on the filer by its queued
close, and a listed directory serves the source from the cache, so a link
right after close updated a name the filer did not have yet and failed
with EIO.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* mount: skip a link change no newer than the handle's entry

A link change carried into the handle could replace a base already moved
past it by a later acknowledgment. Use the same clock-domain check as
installAckedEntry.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* mount: keep a link change's ctime in the dirty entry

The flush does not restamp times, so a dirty entry carrying an earlier
write's ctime moved the filer's ctime back from the link or unlink.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* mount: flush through the file's current name when the resolved one is gone

doFlush resolves the name before uploading the data. An unlink of that
alias, or a rename, in the meantime sent the publish to a name the file no
longer has.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* mount: keep the pending link read when its name is already gone

A flush could read the removed alias name, find nothing, and drop the
pending read of the surviving link, leaving the stale base in place.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* mount: test link and alias unlink across queued closes, flush races, versions and ctime

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* mount: pick a flush's name again once its locks are held

An unlink or rename that lands between the name pick and the locks would
otherwise send the flush to the removed name, where the create guard fails.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* mount: test a flush waiting on its lock through an alias unlink

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* mount: check the flush target after every lock acquisition

A flush whose name kept changing published to the name chosen before its
last lock wait. Check after each acquisition and return a retryable error
once the attempts run out.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* mount: test a flush whose name changes during every lock wait

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 12:25:50 -08:00
bac2196d65 shell: return flag parse errors from every command (#12618)
* shell: return flag parse errors from every command

48 commands returned nil when their flags failed to parse, so a mistyped
flag printed the flag error but the command still counted as a success:
a piped weed shell run exited 0, the master maintenance loop logged no
error, and admin script jobs reported the line as done.

Return the parse error as s3.bucket.create (#12544) and the other shell
commands already do. TestCommandsReturnFlagParseErrors runs every
registered command with an unknown flag and fails for the 48 on master.

Signed-off-by: anant <170962632+antcybersec@users.noreply.github.com>

* shell: don't report -h as a failure

With the parse errors returned, -h on those commands now returns
flag.ErrHelp, so the shell printed "error: flag: help requested" and a
piped run exited 1. The flag package has already printed the usage, so
processEachCmd now treats flag.ErrHelp as success for every command.

Signed-off-by: anant <170962632+antcybersec@users.noreply.github.com>

* shell: keep the -h test beside the flag parse test

TestProcessEachCmdTreatsHelpAsSuccess covers the same change as
TestCommandsReturnFlagParseErrors, so it lives in the same new file and
shell_liner_test.go stays as it was.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Signed-off-by: anant <170962632+antcybersec@users.noreply.github.com>
Co-authored-by: anant <170962632+antcybersec@users.noreply.github.com>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 12:19:44 -08:00
Chris LuandClaude Opus 5.5 c248ab7865 credential: keep IAM changes made during the legacy-file migration (#12614)
* credential: migrate the legacy IAM file without overwriting newer entries

The migration wrote every identity, service account and group from the
snapshot the load read, over whatever the per-entry files held by then, so
a revoke or delete landing in between was undone for good. It now migrates
only the legacy file's own entries, creates each per-entry file only when
none exists, and waits for the next load when the legacy file changed
after the snapshot.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* credential: test changes landing between the IAM load and its migration

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* credential: keep a per-entry file deleted after the IAM load deleted

An entry the load read from its own file is already migrated. When that
file is removed before the migration runs, the legacy copy must not
recreate it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* credential: test a deleted per-entry file the legacy file also holds

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 12:17:45 -08:00
Chris LuandClaude Opus 5.5 12cb707e58 seaweed-volume: copy master.proto with metrics_port (#12605)
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 12:10:48 -08:00
Chris LuandClaude Opus 5.5 aa901b7ac7 s3tables, iceberg: create missing directories exclusively so a concurrent create keeps its record (#12600)
* s3tables: create a missing directory exclusively in ensureDirectory

ensureDirectory looked the path up and, when it was missing, created it with a
plain CreateEntry. A table create that lost the lookup race then replaced the
directory another create had made, claimed and confirmed since, erasing that
table's record, so a third create could claim the name and win as well.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* iceberg: create missing metadata parent directories exclusively

saveMetadataBlob created each missing directory on the way to the metadata
directory with a plain CreateEntry after a lookup. A table directory another
create had made and written its record onto since the lookup was replaced by a
bare directory, erasing that record.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* iceberg: create missing stage-create marker directories exclusively

The marker path starts at the table bucket directory, which carries the bucket
record; a plain create after a missed lookup replaces a directory another writer
made in between, with whatever it carries.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 12:10:18 -08:00
Chris LuandClaude Opus 5.5 e0beb59ef0 topology: treat a nil ttl as the empty ttl when looking for duplicate volume ids (#12601)
Volume layouts are keyed by ttl.String(), so a nil ttl and the empty ttl
share one layout. ttlEqual told them apart, so a replica reported with a nil
ttl into a layout created with the empty ttl matched its own layout as a
duplicate, and every copy was quarantined.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 12:09:07 -08:00
Chris LuandClaude Opus 5.5 fe14880f4d shell: tier download replicates the downloaded volume before deleting the remote object (#12599)
* shell: replicate the downloaded volume before deleting its remote object

volume.tier.download downloaded every replica from the remote tier and let
the last one in the master's location list delete the shared object, so a
replica missing from that list kept reading a deleted object.

Download onto one replica, copy it over every other remote-backed replica,
re-read the topology so replicas that show up meanwhile are replaced too,
and only then delete the last remote-backed replica together with the
object before copying the local volume onto it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* shell: test tier download replicates before deleting the remote object

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* shell: copy the downloaded volume onto each replica's own disk type

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* shell: test tier download keeps each replica's disk type

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* shell: name the hdd disk type when copying a downloaded volume

VolumeCopy treats an empty disk type as the source's, so an HDD replica
copied from an SSD source was placed on SSD or failed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* shell: block deletes on the tier download copy source

A readonly-can-delete source took deletes after VolumeCopy read its
index, so the replaced replica lost them and served the files again.
Hard-mark the source readonly while replicas are replaced, then restore
the writable mark or reload the persisted state.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* shell: keep the remote object while replicas are unlisted

Tier download deleted the shared object once the listed replicas were
local, even when the replica placement expected more copies than the
topology listed. Keep it in that case so a rerun can finish later.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* shell: fail tier download when no replica is listed

Also point the final-replica delete error at volume.fix.replication.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* shell: keep the remote object when a server lists the volume twice

A stale twin on another disk of a replaced server still references the
object but is filtered out with its server.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* shell: restore the tier download source when its readonly mark fails

VolumeMarkReadonly blocks local writes and deletes before its second master
notification, so a failed notification returns an error with the source
already frozen. Restore it whenever the status was read.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* shell: report a failed tier download source restore

A restore that leaves the source unmounted or frozen was only printed, so the
command still returned success. Join the restore error into the result.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* shell: wrap the tier download restore error

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 12:07:52 -08:00
Chris LuandClaude Opus 5.5 94c3869057 volume_move: keep the live-move source frozen across volume reloads (#12597)
* volume_move: persist the move's source freeze across reloads

A live move froze the source in memory only, so an unmount/mount or a
volume server restart mid-move reopened it for writes that the move then
deletes with the source. Persist the freeze when the move froze a
writable source, and reopen the target right after the copy since the
copied .vif carries the freeze. An already-readonly source keeps an
in-memory mark, and a failed move still restores the source writable.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test: live move keeps the source frozen across a reload, on Go and Rust volume servers

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test: require 409 from the reloaded frozen source

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* volume_move: reopen an interrupted move's source only without a target copy

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 12:05:58 -08:00
Chris LuandClaude Opus 5.5 652648d9c4 mount: merge concurrent content changes from another mount instead of failing or dropping them (#12596)
* mount: merge a refused content flush onto the stored entry

A content flush conditioned on the handle's base failed with EIO whenever a
peer had changed the file since, so two mounts appending to one file could
not both close it cleanly. On a refused condition, reread the entry and
replay this handle's changes since its base onto it: the chunks it added or
dropped, and the attributes, extended keys and inline content it changed.
A local truncation keeps its size and clips the peer's chunks to it;
otherwise the larger size wins. The merged entry is published against the
stored one, up to three times, and installed in the handle so its next
flush keeps the peer's chunks. A replaced or deleted file is still refused.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* mount: merge a peer's content change into a dirty handle

A subscription event carrying a peer's content change replaced a dirty
handle's entry and discarded its unflushed pages, so the handle's writes
were lost and its close still returned OK. Fold the peer's entry into the
handle with the same merge a refused flush uses, keep the dirty pages, and
take the peer's entry as the new base. A replaced file still invalidates.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* mount: stop a flush merge once a filer accepts the write

withFilerClient can call the callback on several filers. A refusal from
one left its stored entry behind, so a later filer's success went on to
another merge round.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* mount: refuse a merge that puts inline content next to chunks

A read inside the inline bytes returns them without looking at chunks,
so one side's inline content merged with the other side's chunks hid
data after a successful flush.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* mount: refuse to merge a whole-file rewrite over a peer's chunk change

maybeMergeChunks re-uploads the handle's whole view as unstamped chunks.
Merged after a peer's change, they tie with a peer's own rewrite or bring
back a range the peer truncated, so that flush fails as before.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* mount: clip peer chunks at the lowest truncate, not the final size

A handle that truncated and then wrote past the old end ends up larger
than its base, so the merge did not see the truncate and kept the peer's
chunks inside the range the truncate cleared. SetAttr now records the
lowest size truncated to over the current base.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* mount: keep a peer's truncate when the handle left the size alone

The merge took the larger of the handle's and the stored size, so a peer
truncate came back as a zero-filled tail when the handle only overwrote
bytes inside it. The stored size now stands unless the handle changed
its size or wrote past it, and rebasing keeps only the growth writes made
after the flush was taken.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* mount: retry a read view whose merged manifests did not resolve

A failed SetChunks leaves the chunk group answering every read with the
resolution error, and nothing re-ran it while the entry stayed the same.
The handle now marks such a view (on rebase and on SetEntry) and the
next read or lseek resolves the entry's chunks again.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* mount: keep a dirty handle's writes when a peer change cannot merge

The new merge refusals sent a peer event on the same file down the old
path, which discarded the dirty pages and adopted the peer's entry, so
close returned OK without the writes. The handle now keeps its writes and
base, and the flush meets the conflict and fails.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* mount: keep an unpublished truncate across a peer's base change

The truncate mark was tied to the base pointer, so a peer event that moved
a dirty handle's base (a merged content change or a metadata-only change)
dropped it, and a later merge on a regrown file kept peer chunks inside
the truncated range. Those base moves now carry the mark; a flush ack or a
discarded handle state still clears it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* mount: wait for the flush before merging a shrink over buffered writes

A write inside the file leaves its bytes in dirty pages without moving
the size. Merging a peer's truncate then adopted the smaller size, and the
chunk uploaded later sat past it, unreadable through the handle. The event
path and the periodic metadata flush now refuse a merge that shrinks the
handle; the close flush uploads the pages first and merges then.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* mount: tell a shortened base chunk from a whole-file rewrite

A truncate shortens a base chunk in place, so it gets a new reference
key, and an old chunk can carry no timestamp. The rewrite refusal now
applies only to unstamped chunks whose file id is new to the base.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* mount: check the creation time when the base has no inode

An entry without an inode or hard link id matched any replacement at the
same path, so a stale handle could merge into a different file. Without
an inode the merge now needs the same creation time. The filer keeps it
across updates; only a synthesized base has neither and still merges.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* mount: keep a dirty handle's writes when a peer replaces the file

A peer entry with another inode at the handle's path went down the
fallback that destroys the dirty pages and installs the replacement, so
close returned OK without the writes. The handle now keeps its writes
and base like it does for an unmergeable change, and its flush refuses to
update the replacement.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* mount: build the content flush tests on freebsd

The invalidation tests, which every platform compiles, now use the
content flush helpers, so the freebsd cross-compile vet failed with
undefined: flushConditionEntry. Nothing in the file is freebsd-specific.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* mount: keep a dirty handle's base when a same-content file replaces it

A replacement with the base's chunks but another inode counted as a
metadata-only change, so the dirty handle adopted it as its base and the
next conditional flush overwrote the new file.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* mount: bring a peer's metadata change into a dirty handle

A metadata-only event moved a dirty handle's base but left its live
entry, so the next flush read the peer's chmod as a local edit and
published the old mode back. Replay the handle's own changes onto the
peer's entry instead, as for a content change.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* mount: wait for the flush before merging peer inline content over buffered writes

A buffered write is not a chunk yet, so a merge could install a peer's
inline content that later reads return ahead of the write's chunk.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* mount: track a whole-file chunk rewrite on the handle

The merge told a rewrite by its unstamped chunks, which a manifest fold
hides: manifests are unstamped either way. Record the rewrite against the
base it was made from, carried across peer base changes like a pending
truncate, and refuse the merge from that.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* mount: refuse a merge that shortens a manifest at the truncate

Clipping a manifest reference leaves its members whole, so peer bytes past
the handle's truncate could reappear once the file grew again.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* mount: flush a dirty handle after a peer replaces its file

The replacement test checked only the retained handle state. It now
flushes against a filer holding the replacement, including one with the
base's content, and checks the flush fails and leaves it unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* mount: test an append fsync after a peer's attribute-only change

A second mount's acknowledged chmod, chown, utimes or setxattr leaves
the content alone, so the writer's conditional content flush must still
publish its append and keep that change. On master the refused flush
surfaces as EIO from Fsync.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 12:02:42 -08:00
Chris LuandClaude Opus 5.5 a576398662 weed-vector: advertise its metrics port when registering (#12604)
* master.proto: KeepConnectedRequest carries the node's metrics port

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* weed-vector: advertise --metricsPort when registering with the master

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* master.proto: number metrics_port 9

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 22:36:02 -08:00
Chris LuandClaude Opus 5.5 baaa60b24a admin: an arrow icon for vector buckets (#12602)
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 22:11:13 -08:00
Chris LuandClaude Opus 5.5 917cf534db s3api: sign every operation subresource in signature V2 (#12575)
* s3api: sign every operation subresource in signature V2

The V2 canonical resource only kept the subresources inherited from the
original allowlist, so tagging, retention, legal-hold, cors, encryption,
object-lock and the other keys the router dispatches on were left out of
the signature. A V2 signature issued for one request therefore also
verified that request with one of those keys appended, running a
different operation under the signer's identity.

Add every operation-selecting subresource to the signed list. Clients
that already sign tagging, cors, object-lock or replication in V2 now
verify as well.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3api: sign every value of a repeated signature V2 subresource

canonicalizedResourceV2 kept only the last value of a repeated query key,
while handlers read the first. A request signed for versionId=A could be
sent as versionId=B&versionId=A and act on version B. Sign every value in
order, as V2 clients do, so a second value no longer verifies.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 17:22:59 -08:00
github-actions[bot] 5ba77af65f docs: regenerate star history chart 2026-11-13 01:20:00 +00:00
c2b2ed769b fix(mount): preserve appended chunks during stale attribute updates (#12593)
* fix(mount): preserve acknowledged chunks during attribute updates

* mount: keep the WFS field block aligned

Adding the publication-lock fields inside the aligned block made gofmt
re-indent 25 unrelated lines. Put them in their own group after
dirIdleEvict, matching the enterprise layout, so syncs do not conflict there.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* mount: finish attribute saves through a FUSE interrupt

saveEntryAttributes derived its RPC context from the syscall's cancel
channel, so an interrupt while the conditional UpdateEntry was at the
owner returned EIO for an update the owner then applied, and skipped the
local install and retention. Keep the metadata deadline but drop the
cancellation, like the other metadata mutations.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* mount: map attribute CAS ids in place instead of cloning the entry

saveEntryAttributes cloned the whole entry three times per attempt only
to rewrite uid/gid for the filer, and each clone is as large as the
chunk list. Switch the ids in place and restore the exact local values
after the call, keeping one clone. The test now checks the reply carries
the local uid.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* mount: compare attribute updates against the filer's own uid/gid

With -map.uid/-map.gid, a filer id that is also a mapped local id reads
back unchanged and maps forward to another filer id. The IF_ENTRY_EQUAL
expected entry then never matched, so chmod, utimes and xattr changes on
such files failed with EAGAIN after three attempts, and an update would
have rewritten the owner. Keep the filer ids read with the entry for the
ids the syscall leaves alone.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(mount): align attribute retention with read lock order

* mount: take the handle lock before entryLock in the attribute snapshot

attributeEntrySnapshot held entryLock while it queued for the shared
handle lock. A Read holding that lock waits on entryLock, the snapshot
waits behind a queued exclusive (an invalidation or a Write), and the
exclusive waits on the Read.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(mount): complete handle lock order and preserve explicit ID intent

* mount: re-read a refused attribute update from the file's owner filer

The filer checks the IF_ENTRY_EQUAL condition on the owner of the path's
write route key. The retry read any filer, so behind one that had not
applied the owner's latest update, every attempt sent the same stale base
and the syscall failed with EAGAIN. Read from the owner once the lock ring
is known, like lookupFilerEntry, but keep the filer's raw ids.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 16:28:47 -08:00
d7ae08ecad fix(mount): refuse stale whole-entry content flushes (#12594)
* fix(mount): condition content flushes on the observed entry

* mount: stop retrying a flush the filer refused on its condition

The filer reports a failed CreateEntry condition as FilerError_PRECONDITION_FAILED
in the response, which surfaces as filer_pb.ErrPreconditionFailed rather than a
FailedPrecondition status. The flush retry loop treated it as transient and
replayed the refused flush three more times over 7 s before returning EIO.

The test filer now answers like the real one.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* mount: take the handle base from the entry the filer stored

After an acknowledged flush the handle base was the requested entry, but the
filer stores it with a fresh commit_ts_ns (and fills crtime, inode, atime), so
the next IF_ENTRY_EQUAL flush on the same handle never matched: write, fsync,
write, fsync returned EIO, as did close after a periodic metadata flush and the
first flush after an eager create. Use the stored entry from the
acknowledgement's metadata event.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* mount: condition the periodic metadata flush like the close flush

The periodic metadata flush of an open, dirty handle sent the same whole entry
through an unconditional CreateEntry, so a file held open past the flush
interval still overwrote a peer's acknowledged append, and the close flush then
matched the overwritten state. Both paths now publish through one helper that
carries the handle's write condition.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* mount: republish a refused flush once when the stored entry is unchanged

A handle's base can differ from the stored entry without any peer write: a
rename while the file is open restamps the entry under its new name, and with
-map.uid a filer id that is also a mapped local id does not map back to itself.
Each made every flush of the handle fail with EIO. On a miss, read the stored
entry and, if it still holds the observed state apart from name, commit stamp,
id encoding and derived size, retry once with that entry as the condition.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* mount: take the filer's entry once a write open holds the DLM lock

The lock orders this writer after the previous holder, but the holder's
flush may not have reached the local cache yet, so the handle's base
named the entry from before the wait and the conditional flush refused.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* mount: share the open-existing path of Create

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* mount: open the file a previous DLM holder created during a deferred create

A deferred create never asks the filer, so a create that waited for the
lock went on to publish over a file the previous holder had just made,
and its flush expecting no entry was refused.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* master: send the lock ring snapshot to mounts

A mount running -dlm is a lock client like the S3 gateway, and needs the
ring on connect to reach a file's owner filer before the next membership
change is broadcast.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* mount: follow the masters' lock ring under -dlm

The lock client only knew the seed filer, so every lock took the forward
hop and the mount had no way to tell which filer owns a file. Subscribe to
the masters named by the filer, like the S3 gateway, and wait briefly for
the first ring before serving.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* mount: read the DLM-ordered entry from the file's owner filer

The filer checks a flush's condition on the owner of the path's write
route key, and another filer may not have applied the previous holder's
flush yet. Look the entry up on that owner once the ring is known.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 16:15:08 -08:00
Chris LuandClaude Opus 5.5 a2b9b37ee5 volume(rust): untrack queued writes before replying (#12595)
* volume(rust): untrack queued writes before replying

The durable write worker sent each write's result and only then dropped
the batch's pending-write trackers, so a caller could see its write
finish while the write was still tracked as pending. Go untracks in the
caller before writeNeedle2 returns. Drop the trackers before sending the
results so the Rust queue path matches.

test_replicate_delete_cancels_a_queued_write asserted the count was zero
right after the fsync write returned and failed about 1 run in 100 under
load (Rust Unit Tests on master c2625dd).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* volume(rust): untrack a write rejected for a remounted volume before replying

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 16:11:10 -08:00
Chris LuandClaude Opus 5.5 a2ffdce2ec volume: skip VolumeDelete when its RPC was cancelled before the delete (#12586)
* volume: skip VolumeDelete when its RPC was cancelled before the delete

A caller that cancels (a lost tiering lease, an expired deadline) still
had its admitted delete run in the Go server, including one that waited
behind the copy guard. The Rust server drops a cancelled handler, so
return the context error before the delete starts.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* volume: wait until the queued delete blocks on the guard before cancelling

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 10:19:05 -08:00
Chris LuandClaude Opus 5.5 6388dfca9b s3api: verify signed request bodies against the signed payload hash (#12577)
* s3api: verify signed request bodies against the signed payload hash

A SigV4 signature covers the declared x-amz-content-sha256, but only
getRequestDataReader (PutObject, UploadPart) compared the body with it.
Every other handler parsed the body as received, so a signed request,
DeleteObjects for example, could be replayed with a different body.

After a V4 signature verifies, wrap the body so it is handed out only
once all of it has been read and matched against the hash the signature
covered; a decoder that stops reading early never sees unverified bytes.
getRequestDataReader unwraps it and keeps verifying uploads as they
stream.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3api: answer XAmzContentSHA256Mismatch for refused DeleteObjects bodies

DeleteObjects answered a body that failed its payload-hash check with
InternalError and CompleteMultipartUpload with MalformedXML. Name the
check that failed instead, as PutObject does.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3api: check Content-MD5 on signed request bodies

Signature V2 signs Content-MD5 rather than a payload hash, and a V4
request sent with UNSIGNED-PAYLOAD may still sign it, but bodies read
outside the upload path never compared it with the body. Check it with
the payload hash once a V2 or V4 signature verifies: BadDigest on a
mismatch, InvalidDigest when it is malformed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3api: verify presigned uploads against a signed payload hash

A presigned V4 URL can sign X-Amz-Content-Sha256 as a query parameter,
but getRequestDataReader compared upload bodies only with the header of
that name, so the signed hash went unchecked. Verify uploads against the
hash the signature covered.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3api: check Content-MD5 on bodies buffered for their payload hash

IAM and other non-S3 requests signed without x-amz-content-sha256 have
their body buffered and hashed during signature verification. That
buffer counted as already bound, so their Content-MD5 was never checked.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3api: check Content-MD5 on directory marker bodies

A PUT of a key ending in "/" stores its body on the directory entry
without going through putToFiler, so its Content-MD5 was never compared
with the body. Answer BadDigest on a mismatch, as putToFiler does.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3api: wrap and unwrap signed bodies through helpers

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 10:17:09 -08:00
Chris LuandClaude Opus 5.5 a3b68aa37c shell: confirm trim survivors still serve the volume before deleting (#12590)
* shell: confirm trim survivors still serve the volume before deleting

volume.fix.replication compared the doomed replica's index with every
survivor and then deleted it unconditionally. A survivor that went away
while the indexes were read still let the delete through, leaving fewer
copies than the check had counted. Re-read each survivor's VolumeStatus
after the index checks and skip the trim if any no longer has the volume.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* shell: test a trim whose survivor goes away after its index read

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 09:32:37 -08:00
Chris LuandClaude Opus 5.5 4cba4eb7bf s3: ignore unsigned object headers on POST Object (#12585)
* s3: ignore unsigned object headers on POST Object

The POST policy signs form fields only, but the upload read tagging,
user metadata, storage class, object lock and the stored content
headers from the request headers as well. Drop them before the form
fields are applied.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3: test POST Object request headers against signed form fields

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 09:28:33 -08:00
Chris LuandClaude Opus 5.5 249f04e63a s3api: NoSuchKey for object lock calls on missing or deleted keys (#12592)
* s3api: answer PutObjectLegalHold on a missing key with NoSuchKey

In a versioned bucket a key with no latest version resolves to
ErrLatestVersionNotFound, which the legal hold handler did not map, so
the request failed with 500 InternalError. PutObjectRetention already
maps it to NoSuchKey.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3api: keep failed reads distinct when resolving an object lock target

getObjectEntryForUpdate turned every lookup error into a not-found
sentinel, so a failed filer read made PutObjectRetention and
PutObjectLegalHold answer NoSuchKey for an object that exists. Only a
genuine miss becomes the sentinel now, matching getObjectEntry.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3api: answer object lock calls on a delete marker with NoSuchKey

PutObjectLegalHold and PutObjectRetention on a key whose current
version is a delete marker returned 200 and wrote the lock onto the
marker, and the matching GETs read it back. A key whose current version
is a delete marker does not exist, and delete markers take no lock, so
these calls now answer NoSuchKey like the tagging and ACL handlers.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3api: let a lock recorded on a delete marker not block its removal

Delete markers are not WORM-protected, but a legal hold or retention
written onto one (possible before the previous change) made both the
gateway check and the routed delete's filer-side WORM guard refuse to
remove the marker, so the object could not be restored and lifecycle
could not clean the expired marker up.

Lock enforcement now ignores a lock on a delete marker, and the routed
version deletes in DeleteObject and lifecycle drop the WORM guard when
the version they read is a marker. A version id never changes kind, so
the marker read is the one deleted; the null version keeps its guard
because a PUT can replace it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 09:27:12 -08:00
Chris LuandClaude Opus 5.5 4ee82977af docker: pass entrypoint arguments to weed without splitting them (#12589)
The master, volume, volume-rust, server, mini, filer and s3 cases and the
fallback forwarded the user arguments as an unquoted $@, so the shell split
each one at whitespace and globbed it against /data. An -admin.password with
a space reached weed as two words, and weed ignored every flag after the
stray word. Quote "$@" as the other cases already do; $ARGS stays unquoted
since the script builds it from its own words.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 09:25:25 -08:00
Chris LuandClaude Opus 5.5 bef1b13b3d replication: keep a local sink path the source recreated after a delete (#12581)
* replication: let the local sink check a delete against the live source

LocalSink removed the destination path for any delete it was handed. A
delete that reaches the sink after the source recreated the path (a peer
filer's event delivered late, or a historical event replayed) removed the
newer file, and no later event brings it back.

DeleteEntryVersion receives the deleted entry. Before removing an existing
destination, the local sink looks up the source path and keeps the
destination when the source holds an entry of the same kind committed
after the deleted version (commit_ts_ns, or mtime for unstamped entries).
A live entry that is the deleted version itself or older means the source
store lags the event, so the delete proceeds. A failed lookup returns the
error so the event is retried instead of deleting a file that may be live.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* filer.sync, filer.replicate: hand the deleted version to the sink

Pure deletes, renames without a native move and moves out of the
watched directory now go through sink.DeleteEntryVersion with the old
entry, so a sink that checks the live source (the local sink) can keep a
path the source recreated after the deleted version. Other sinks fall
back to DeleteEntry unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* replication: order the local sink check by the delete's mutation time

A source filer can hold a version written concurrently with the delete:
newer than the deleted entry but older than the delete itself. The
source's own replay applies the delete over it, so keeping the backup
copy because it outranks the deleted entry left a file the source no
longer has.

The sink now gets the delete's mutation time (MutationTsNs, else TsNs)
and keeps the destination only when the live entry was committed after
it, the order Filer.Replay keeps. That also covers a stamped replacement
of an unstamped entry whose mtime did not move. filer.replicate events
carry no time, so they keep the deleted entry's stamp as the bound, and
mtime only when nothing is stamped.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* replication: count an unstamped live entry from its mtime

A source filer that predates commit stamps stores entries with no
CommitTsNs, so comparing the live stamp with the delete's time never
kept a replacement there. An unstamped live entry now counts from its
mtime against the delete's time.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* replication: keep a stamped live entry when the deleted one is unstamped and the delete has no time

filer.replicate gets no delete time from the notification queue. When the
deleted entry has no commit stamp either, a stamped live entry was written
after it, so its mtime no longer decides.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 09:23:54 -08:00
Chris LuandClaude Opus 5.5 c2625ddf00 seaweed-volume: read security.toml values as TOML strings (#12582)
The line parser only trimmed quotes, so `key = "abc\n"` gave the Rust
volume server the bytes `abc\n` with a literal backslash while the Go
servers sign with a newline, and every JWT failed. Literal strings kept
their quotes and a trailing comment stayed part of the value.

Decode each value with the TOML parser; text it rejects keeps the old
quote trimming.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 09:19:18 -08:00
Chris LuandClaude Opus 5.5 b7d6b48921 operation: probe octet-stream uploads for gzip like untyped ones (#12584)
* operation: probe octet-stream uploads for gzip like untyped ones

The 128-byte gzip probe ran only when the caller passed no MimeType, so an
upload labelled application/octet-stream or binary/octet-stream was never
compressed. S3 PutObject and UploadPart pass the request Content-Type (or the
octet-stream that mimeDetect fills in), so every binary S3 upload skipped
compression that the same bytes get through the filer.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* operation: check the probed upload keeps the caller's content type

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 09:16:55 -08:00
Chris LuandClaude Opus 5.5 02b9747bd8 wdclient: keep a shared volume lookup alive when its first caller leaves (#12579)
* wdclient: keep a shared volume lookup alive when its first caller leaves

Concurrent misses for the same volumes share one provider lookup, but it
ran on the first caller's context, so that caller's cancellation failed
every request waiting on it. The lookup now runs on a context no single
caller can cancel, and each caller stops waiting when its own context ends.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* wdclient: wait for the second caller to join before canceling the first

A fixed sleep did not prove the second caller had joined the shared lookup:
a late caller could hit the filled cache and pass without waiting. The test
now waits until the caller reads its own Done channel, which happens only
after DoChan registered it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 09:15:01 -08:00
Chris LuandClaude Opus 5.5 8581151376 filer: gzip metadata log pieces before uploading them (#12580)
* filer: let the fake-master test helper serve any master

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* filer: gzip metadata log pieces before uploading them

Every metadata event carries the old and new entry with their whole chunk
lists, so a file grown by small fsynced appends logs the same chunk records
again and again: 2000 appends of 256 bytes through weed mount wrote 34.8 MB
of log. The uploader only compresses when the first 128 bytes compress, and
a log piece starts with fresh file ids and ETags, so it stored the log as is.

The filer now gzips each piece itself and keeps the compressed bytes when
they save at least 10%. After the same workload the volumes hold 3.05 MB
instead of 35.2 MB, file data included. Readers already honor the chunk's
IsCompressed flag on every path (filer-side decode, log file refs read
straight from volume servers), and the assign request sizes the bytes
actually stored.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 09:13:50 -08:00
Chris LuandClaude Opus 5.5 e0f8438284 mount: retry a writeback metadata flush until the filer answers (#12578)
* mount: keep the filer status in the flush create error

flushMetadataToFiler formatted the CreateEntry error with %v, so callers
classifying it saw only the message, which includes the file path. Wrap
it so the gRPC status the filer sent stays reachable.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* mount: retry a writeback metadata flush until the filer answers

With -writebackCache, close() returns before the chunk references reach
the filer. The deferred flush gave up after four attempts, about a minute
once each attempt's own filer retries are counted, then released the
handle: a longer filer outage dropped the file even though write() and
close() had succeeded, and its uploaded chunks became orphans.

Keep retrying for as long as the filer is unreachable, as the writeback
chunk uploads already do. A filer that answers with an error still ends
the flush after the bounded attempts, and an unlink of the draining file
still ends it on the next attempt.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* mount: retry only an unanswered filer call without a bound

The unbounded phase used util.IsTransientError, which falls back to
matching the error text. A refusal the filer sends in its response
(read-only, name too long, a plain error string) has no gRPC status, and
the flush error carries the file path, so a file named like
"unavailable-transport.log" kept retrying against a filer that answered.

Retry without a bound only when the call failed with a gRPC Unavailable
status: the dial, the connection or the mount's call deadline gave out.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 08:57:42 -08:00
Chris LuandClaude Opus 5.5 a6a3bdef8d s3api: deny signed reads of a version file through its physical path (#12574)
* s3api: deny signed reads that name a version file by its physical path

A stored version lives at <key>.versions/v_<id>, and that path is also a
valid object key. Anonymous reads already refuse it; signed GET, HEAD,
copy sources and object subresources did not, so a caller without
s3:GetObjectVersion, or under a Deny on the logical key, could read any
version through it. Apply the same check in the shared entry recheck.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3api: recheck a version file key before answering a conditional read

A conditional GET or HEAD answered 304 or 412 with the ETag before the
handler's entry recheck, so the physical version path still confirmed a
version's ETag to signed callers.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3api: look up version file keys in the lookup-based policy recheck

Retention, legal hold, tagging writes and deletes only fetched the entry
for the recheck when the bucket had a policy, so without one they still
acted on a version file named by its physical path. Fetch it whenever
the key has the version file shape.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3api: refuse a directory-marker delete that names a version file

A trailing slash kept "<key>.versions/v_<id>/" out of the version file
shape check, and the lookup recheck misses the entry for that form, so a
DELETE reached deleteDirectoryMarker and removed a chunkless version or
delete marker without s3:DeleteObjectVersion.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 08:21:31 -08:00
Chris LuandClaude Opus 5.5 317d907334 filer: route tests forward to a peer port the test holds (#12576)
The ring tests used 127.0.0.1:18889 as an unreachable peer, but its gRPC
port 28889 is a common filer port: on a shared CI host a live filer there
accepted the forwarded exclusive create, and
TestCreateEntryExclusiveLeavesNonOwner failed. The peer now listens on a
port the test owns and drops every connection.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-11 19:10:58 -08:00
github-actions[bot] d51a79335c docs: regenerate star history chart 2026-11-12 00:47:35 +00:00
Chris LuandClaude Opus 5.5 13334a59b9 vector: reuse the Lance session, version leases and opened tables across reads (#12573)
* vector: share one Lance session across reads

Each request opened its table in a session of its own, so every query
loaded and decoded the index again. Get, list and query now open in one
session the server keeps, sized by -index-cache-mb and
-metadata-cache-mb; writes keep a session of their own.

Lance keys some of its cached entries, such as a version's index list,
by location and version alone, and an index dropped and created again at
its location numbers its versions from 1 again: a shared session answered
a query on the new table with the old table's index. CreateIndex now
marks each table with an id in its schema metadata, and the session
starts over when a location turns up holding another table.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* vector: share a version lease among the reads of that version

Every get, list and query tagged the version it read and deleted the tag
afterwards: two filer writes and a manifest read per request. Reads of
one version now share one tag, which goes 10 s after the last read took
it; a tag is reused for at most 5 minutes, so a busy version never
reaches the reapers' TTLs.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* vector: keep opened tables for the next read

A read built the table's store and read its latest manifest each time.
Reads now keep the opened table by location and encryption, up to 1024,
and move it to the latest version, which lists the versions and takes
the manifest from the session. A handle opened in an earlier session is
not reused.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* vector: key shared leases by manifest, and take one per version at once

A table dropped and created again within the linger reused, for a
version number the old table had, a lease whose tag the drop had
deleted. Leases are now keyed by the manifest's ETag as well. Reads that
arrive before a version has a lease wait for the first to take it,
rather than each writing a tag.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* vector: drop kept tables when the shared session starts over

A kept table holds the session it was opened in, so a replaced session
and its caches lived on in the handles of other tables until they were
evicted.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* vector: prune shared lease slots only when they double

Every lease lookup scanned all slots under the map's lock, a cost that
grew with the versions read in the last linger.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* vector: shrink the lease slot map after a prune

After a burst the map kept its peak allocation, which every later prune
scanned.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* vector: keep a lease slot a read has taken through a prune

A read that took a slot and had not yet locked it could see the slot
pruned, and then pin through it while the next read pinned through a
new slot, writing a second tag.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-11 16:26:50 -08:00
Chris LuandClaude Opus 5.5 b924ffabf1 admin: calm console restyle with light and dark themes (#12568)
* admin: restyle the UI as a calm console

Grey canvas with white sheets, a docked sidebar that marks the current
page with an indigo tint, ink primary buttons, tinted status pills,
dense tables with tabular numbers and spectrum capacity meters. The
styles are design tokens layered over Bootstrap's own variables, with
light and dark values, so the page templates stay unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* admin: add a light, dark and automatic theme switch

The choice is kept in localStorage and applied before the page paints,
on the sign-in page too; automatic follows the system setting.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* admin: drop hard-coded colors from page templates

The volume id link, utilization track and plugin form borders used
fixed light-theme colors; sparklines now use the data palette.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* admin: theme the plugin job detail panel background

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* admin: keep the selected radio dot visible in dark mode

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* admin: cycle the theme from memory when storage is blocked

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* admin: keep success progress bars green

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* admin: keep keyboard focus visible on the toolbar togglers

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* admin: use contrasting text on danger and success buttons in dark mode

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* admin: lead the dashboard with cluster health and one row of tiles

The header now says in one line whether the cluster is healthy and how
many masters, volume servers and filers report. The two rows of stat
tiles become one: the volume size limit and EC volume count move into
footnotes, the mount client card joins the cluster counts, and the
counts link to their pages instead of sitting in nested cards. The
duplicate version list and the bucket shortcut in the header are gone,
and the section menus become plain View all links.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* admin: export the user columns the table actually has

The users table has username, email and access key columns, but the
export waited for six cells per row and so always produced only the
header line.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* admin: drop table titles that repeat the page title

List pages titled their only table with the page title again. The
table now sits right under the summary tiles, and the export actions
that were hidden in the table header menu move to the page header as
an Export button, matching the cluster pages. The policies menu linked
to nothing and is removed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* admin: drop clock and duplicate summary tiles

Buckets, users, policies and service accounts spent a summary tile on
the refresh time, which the page footer already shows (policies gains
that footer). The users page also showed its user count twice.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* admin: show unset bucket settings as a dash

Rows of No owner, No quota and Not configured made the bucket table
hard to scan. Unset cells now show a quiet dash that keeps the wording
as a tooltip and for screen readers, and the name and creation date no
longer wrap (the full timestamp is in the date's tooltip).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* admin: export buckets through the page's API exporter

admin.js loads after the page, so its copy of exportBucketList replaced
the buckets page's own one. That copy scraped the visible rows, so it
exported only the current page, without CSV quoting, and picked up the
display text of the cells. The page's exporter reads every bucket from
the API with full timestamps and escaped fields.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* admin: escape quotes in the user export

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* admin: download the bucket export as a Blob

The exporter put the CSV into a data URL with encodeURI, which leaves #
alone, so a bucket owner containing # ended the download there. Use the
shared downloadCSV helper instead.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* admin: count masters as configured, not reporting

The master list comes from raft membership, which keeps a follower
that is down, so it does not show which masters are reporting.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* admin: link the Prometheus metrics endpoint from the sidebar

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* admin: drop the sidebar metrics link

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-11 15:32:48 -08:00
Chris LuandClaude Opus 5.5 1618d613a3 s3: resolve a role session's aws:username from its user, not its session name (#12572)
* s3api: record the caller on AssumeRole sessions

The session name is the caller's choice; the session now carries the
caller itself as the user it was issued to, as the web identity, LDAP
and credential paths already do.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3api: take a role session's aws:username from its user, not its name

An assumed-role ARN ends in the session name the caller picked, so
naming a session after another user resolved ${aws:username} and
${aws:userid} to that user in bucket and attached policies. Resolve
them from the user the session was issued to instead.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* iam: take a role session's aws:username from its user, not its name

IsActionAllowed set aws:username to the last segment of an assumed-role
ARN, the caller-chosen session name, for role and bucket policies. Use
the user the session was issued to, and drop a forwarded value when the
session names none.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3api: pass a role session caller's user on to the session it assumes

A role session that calls AssumeRole is named by its session ID, so the
new session recorded that ID as its user. An STS session caller now
passes on its own recorded user, or none.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3api: size the session claims map from the request context

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-11 15:32:13 -08:00
Chris LuandClaude Opus 5.5 3f557b73c8 admin: show the right master ports when -port.grpc is not port+10000 (#12570)
* admin: take master HTTP ports from raft member names

The masters page and dashboard turned each raft member's gRPC address
into an HTTP address by subtracting 10000 from the port, so masters with
a custom -port.grpc showed wrong or negative ports. Raft members are
named after their master address, which carries the HTTP port; use it
and keep the old derivation only for names that are not addresses.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* admin: test raft member address mapping

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-11 15:22:29 -08:00
Chris LuandClaude Opus 5.5 a0f95e16bc s3api: fix TestRoutedMultipartFinalize flake from a reused fake filer port (#12571)
* s3api: drop cached gRPC channels when a fake filer takes a port

A test that dials a dead owner leaves its failed channel in the
process-wide gRPC connection cache. When a later test's fake filer is
given the same port, its first RPC fails at once with the cached
connection refused, which is how TestRoutedMultipartFinalize failed on
the 32-bit job. The fake filer helpers now share fakeFilerAddress, which
invalidates any cached channel for the new address.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3api: test a fake filer on a dead owner's reused port

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3api: drop cached channels for dead test filer addresses too

closedFilerAddress can be handed a port whose cached channel belonged to
a fake filer that just stopped and may still look ready; then the dead
owner's transaction is sent on it instead of failing over. Build its
address with fakeFilerAddress as well.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-11 14:50:16 -08:00
Chris LuandClaude Opus 5.5 6c1eec8703 s3: store the POST Object acl on the uploaded object (#12567)
* s3api: split upload ACL assembly out of preparePutObjectACL

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3api: store POST Object acl and grant fields on the object

The form acl and x-amz-grant-* fields were authorized but never assembled,
so the object kept no ACL: acl=private on a public-read bucket stayed
anonymously readable and invalid values were accepted. Build them with the
same rules as PutObject, which also covers the Block Public Access check.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3api: test POST Object stores its upload ACL

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3api: take the POST Object ACL from the signed form only

ACL request headers are not covered by the POST policy, so a holder of a
signed form could add one and have it stored with the upload.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3api: reject a POST acl field the policy does not cover

The acl form field now sets the stored object ACL, so like the x-amz-*
fields it needs a policy condition, as AWS requires.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3api: test unsigned ACL headers and uncovered acl fields on POST

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3api: evaluate POST upload policies on the canonical ACL

Policy conditions on the grant headers saw the raw form value, so an
equivalent spelling of a denied grantee passed. Use the PutObject request
shape that the PUT path evaluates.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3api: check bucket policy for POST ACLs with auth disabled

PutObject applies the bucket policy to an explicit upload ACL when IAM is
off; POST now does the same before storing it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3api: reject differing acl and x-amz-acl POST fields

Both promote to the canned ACL header, so map order picked the stored ACL.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3api: test POST grant canonicalization, auth-off ACL policy, acl conflicts

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3api: check the auth-off POST ACL policy after loading the bucket

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3api: reject an empty acl or x-amz-acl POST field beside a differing one

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3api: test the auth-off POST ACL policy on a cold bucket cache and empty acl aliases

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3api: load the bucket before per-key object authorization

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3api: test the POST ACL policy on a cold bucket cache with auth on

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3api: stop per-key object authorization when the bucket fails to load

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3api: test the cold-bucket POST ACL allow path and a failed bucket load

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-11 10:36:51 -08:00
Chris LuandClaude Opus 5.5 eb159f3181 lance: deregister and re-register need UpdateTable (#12552)
* lance: require UpdateTable to deregister a table

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* lance: require UpdateTable to register over an existing table

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* lance: test that GetTable alone cannot deregister or re-register a table

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* lance: a register that finds a table the caller cannot read needs UpdateTable to bring it back

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* lance: decide declare write access by whether the table existed, not by what the caller can read

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* lance: test register and declare by a caller that cannot read the table

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* lance: register clears the marker on the create branch only after checking it

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* lance: keep a register whose table was created when the marker check fails

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-11 10:35:00 -08:00
Chris LuandClaude Opus 5.5 4cb52d7705 filer: reopen a leveldb store wedged by a transient write error (#12565)
* filer/leveldb_handle: reopen a database whose journal latched a write error

goleveldb keeps the first journal write error and returns it for every
later write until the database is reopened, so one transient ENOSPC or
EIO fails all writes until a restart. The handle offers the leveldb.DB
methods the stores use; a write that fails on an OS error probes the
storage and, once it accepts writes and no call holds the database,
closes and reopens it and retries the write once.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* filer: open the leveldb stores through leveldb_handle

leveldb, leveldb2 and leveldb3 now recover from a transient storage
write error without a restart.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* filer: leveldb stores return iterator errors from listings and deletes

A listing on a database that cannot be read reported an empty
directory, and DeleteFolderChildren wrote an empty batch, because the
iterator error was never checked.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* filer/leveldb_handle: test a reopen that fails after the probe passes

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* filer/leveldb_handle: probe storage before draining readers

A probe that stalls in Create, Write or Sync held mu exclusively and
blocked every read on the handle. Reopen attempts are now serialized by
their own mutex, and the probe runs before the drain.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* filer/leveldb_handle: hold new calls back while a reopen drains

TryLock succeeds only at an instant with no reader, so overlapping
listings could keep a reopen from ever draining while each write spun
for the full drain timeout. The drain now holds new calls back for at
most drainTimeout; a call nested inside a running one waits that out
instead of deadlocking. A drain that times out holds off the next
attempt for busyReopenInterval, so new calls are not held back over and
over.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* filer/leveldb_handle: bound the wait for a running reopen

A write that hit the latched error waited on the reopen mutex for as long
as another caller's storage probe ran, so a stalled probe queued every
writer. Callers now wait at most reopenWait and return their error, and
still retry once a concurrent reopen succeeds.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* filer/leveldb_handle: tests wait for the drain and readers explicitly

The nested-read test polled for a drain that could start and end between
polls and then spun forever; it now syncs on the probe, widens the drain
and fails if the put returns first. The overlapping-readers test waits
until every reader holds an iterator before it writes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-11 10:23:19 -08:00
Chris LuandClaude Opus 5.5 5150168368 s3: authorize an empty versionId as the current-object action (#12566)
* s3: authorize an empty versionId as the current-object action

GET, HEAD and DELETE treat an empty versionId as the current object, but
the action resolver keyed on the parameter's presence and authorized it
as GetObjectVersion/DeleteObjectVersion, so a Deny on GetObject or
DeleteObject did not apply.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3: test an empty versionId against a version-only policy

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3: cover a bare versionId on HEAD

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-11 10:21:28 -08:00
Chris LuandClaude Opus 5.5 0c6c6868de volume: refuse a tier upload whose .dat was written during the copy (#12562)
* volume: refuse a tier upload whose .dat grew during the copy

AttachRemoteDatFile only checked that the data backend was still the one
copied, so a needle write acknowledged after the copy read the .dat was
missing from the remote object, and the upload then switched reads to it
and removed the local .dat. Compare the .dat size with the copy under the
same data-file locks; on a mismatch the handler deletes the copy and keeps
the local file.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* seaweed-volume: refuse a tier upload whose .dat grew during the copy

Mirror the Go AttachRemoteDatFile check. The tier-up attach took the
write lease only inside each step, so a needle write acknowledged after
the upload read the .dat was missing from the object the volume then
switched to, and the local .dat holding it was removed. Hold one write
lease across the instance, revision and size checks, the remote-file
update, the .vif save and the backend swap, as tier-down does; a size
mismatch aborts, deleting the object and keeping the local .dat.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* volume: compare the physical .dat size before a tier attach

DiskFile pads its size past an unaligned tail, which a read-only load
keeps, while the upload copies the physical bytes. Stat the file itself
so such a volume is not refused as written during the copy.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-11 10:19:06 -08:00
dc03c3b1f7 shell: parallel moves and a per-run limit for volumeServer.evacuate (#12561)
* shell: parallel moves and a per-run limit for volumeServer.evacuate

-maxParallelization runs up to N EC shard moves at once (default 1), with the
ec.balance scheduler: never two shards of one volume at a time, free workers
go to the least busy targets. The moves are planned first with the same target
choice as before, then applied through volume_move.Mover.MoveEcShards, which
verifies the target registered a shard before deleting it on the source.

-movesPerExec limits how many volumes and EC shards one run moves (default 0
for unlimited), like -volumesPerExec in volume.balance, so a big server can be
emptied in several runs.

With -skipNonMoveable a shard that fails to move (a read error on a failing
disk, for example) no longer drops the other shards of its volume; they are
moved and the failures are listed at the end. Without it the run stops at the
first failure as before. Losing the admin lock stops the run in either mode.

Refs #12501

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* shell: plan evacuation again after failed moves; fail on a lost lock

A failed shard move kept the target slot it was planned on, so a readable
shard that found no destination in the plan stayed on the server although
the room was never used. Failed moves now give their targets back to the
model and the shards without a destination are planned again, up to
maxEcEvacuationRounds times; the failed shards are not tried again.

With -skipNonMoveable a lost admin lock was printed as a skipped failure and
the run returned success. ec.ErrLockLost is now exported and always fails
the run.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* shell: volumeServer.evacuate reports -movesPerExec after an applied EC run

The round loop only checked the limit after a failed move, so an applied run
that stopped at -movesPerExec without failures never said to run it again.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* shell: volumeServer.evacuate plans again for as long as failed moves leave room

The ten-round cap left readable shards on the server once more than ten
rounds of unreadable shards had taken the free room first, and a rerun
fails the same shards first again. Every round moves or fails each shard it
plans and a failed shard is not planned again, so the rounds end without it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* shell: volumeServer.evacuate gives a failed move back every slot it took

When the target already held the shard in another disk type, adding it to
the planned disk type still took a slot, but the undo gave back only the
explicit one. Record the slots the plan took and restore exactly those.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* ec: an evacuation move that landed on its target keeps the target slot

A move can fail after the target registered the shard, when only the source
unmount or delete failed. Evacuate then gave the target's slot back and could
plan another shard onto a full server. The runner now checks the target after
a failed move and leaves a shard the target has out of FailedMoves.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* shell: volumeServer.evacuate gives back the slot of a failed move onto a shard the target had

A target that held the shard before the plan reports it after any failed
move, so the runner took such a move for one that landed and evacuate kept
the slot the plan had taken. The runner now lists those moves in
LandedMoves, and evacuate undoes the ones whose target had the shard, then
plans again with the room they gave back.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
2026-11-11 10:18:51 -08:00
Chris LuandClaude Opus 5.5 fcfca3fee3 s3, filer: bound the wait for an upload slot (#12559)
* util: add WaitUntil, a cond wait bounded by a context and a timeout

The caller holds the cond lock. Cancellation and the timeout wake the
waiter through a broadcast taken under the same lock, so no wakeup is lost.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3: bound the wait for an upload slot

A PUT that arrived while the concurrent upload limits were exhausted waited
on the condition variable with no bound. The handler reads nothing from the
body while it waits, so a client that gave up and reset its connection went
unnoticed: the socket and goroutine stayed pinned until a slot freed, and a
stalled volume server grew the gateway's fd count by one per aborted PUT.

The wait now ends on InflightUploadDataTimeout with 503 SlowDown, which S3
SDKs retry, and on request cancellation. An admitted upload takes its slot
under the lock, and a finished upload broadcasts, so freed bytes admit every
waiter that fits instead of one.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* filer: bound the wait for an upload slot

The filer's PUT/POST upload limits queued requests the same unbounded way.
Reuse the bounded wait: reject with 429 after InflightUploadDataTimeout, as
the volume server does, and stop on request cancellation.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* command: add -inflightUploadDataTimeout for the S3 gateway and the filer

weed s3 -inflightUploadDataTimeout, weed filer -inflightUploadDataTimeout and
-s3.inflightUploadDataTimeout, weed server and weed mini
-filer.inflightUploadDataTimeout and -s3.inflightUploadDataTimeout. The
default of 60s matches the volume server's flag of the same name; 0 keeps
the old unbounded wait.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3: reject a queued upload without reading its body

WriteErrorResponse drains the request body before answering. Behind the
stats recorder the drain has no read deadline, and reading the body sends an
Expect: 100-continue client the go-ahead to upload what is about to be
discarded. Answer the rejection with an empty-body copy of the request so
the drain is skipped; net/http closes the connection when a large unread
body remains.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3, filer: release an upload slot under the cond lock

A waiter checks the counters under the cond lock and then waits. A release
that decremented and broadcast without the lock could land between the two,
so the waiter slept with a slot free until the next release or its timeout.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3, filer: close the HTTP/1 connection when rejecting a queued upload

net/http reads up to 256 KiB of an unread request body before sending a
keep-alive reply, so a client that stalls its body held back the 503 or
429 until the connection idle timeout. Connection: close skips that read.
HTTP/2 is left alone: there it would shut down the whole connection.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-11 10:17:18 -08:00
Chris LuandClaude Opus 5.5 d659228338 shell: s3.clean.uploads frees an open upload under the object write lock (#12558)
* shell: s3.clean.uploads deletes uploads over filer gRPC

Same DeleteEntry the HTTP handler runs, without minting a filer JWT,
and the gRPC path is the one a locked, routed delete needs next.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* shell: s3.clean.uploads frees an open upload under the object write lock

The cleaner deleted a stale upload recursively with no lock, so a
CompleteMultipartUpload could pass its upload-exists check after the
part entries and their chunks were gone, and publish an object over
freed chunks. Do what AbortMultipartUpload does: hold the object's
distributed write lock, and free an open upload in an ObjectTransaction
routed to the object's owner filer, conditioned on the object not
carrying the upload; if it does, delete the metadata only.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* shell: s3.clean.uploads keeps the object write lock for the whole cleanup

A large upload's recursive delete can outlast one 15 second lease, and
a completion on the distributed-lock path could then take the lock and
commit over chunks being freed. Renew the lock in the background and
skip the delete once it is lost.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* shell: s3.clean.uploads locks the path a completion commits under

For a trailing-slash key a non-versioned completion writes and locks the
entry inside the directory the key names, not the object path, so the
cleaner's transaction did not serialize with it. Lock and check the
same path the completion uses for the bucket's versioning state; the
route key stays the object's.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3api: abort locks the path a completion commits under

A routed abort locked and checked the object path, but for a
trailing-slash key a non-versioned completion commits under the entry
inside the directory the key names. Use the same path the completion
locks for the bucket's versioning state.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* shell: end s3.clean.uploads bucket failure line with a newline

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* cluster: NewBlockingLongLivedLockWithin bounds the wait on a held lock

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* shell: s3.clean.uploads skips an upload whose object lock stays held

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* cluster: share the contended lock filer setup across lock tests

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* shell: s3.clean.uploads reads the bucket's versioning state per upload

The commit path a routed completion locks depends on versioning for a
trailing-slash key. Reading it once per bucket before the run left a
window as long as the run; read it just before the delete, as the
gateway's abort does.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-11 10:16:20 -08:00
Chris LuandClaude Opus 5.5 b11af30cb1 mount: remove the local control socket when the mount exits (#12560)
* mount: remove the local control socket when the mount exits

weed mount listened on /tmp/seaweedfs-mount-<hash>.sock and removed it only
when a later mount of the same directory started, so every unmount or
SIGTERM left one socket file behind. Remove it on the normal return path and
in the interrupt hook, unless a later mount of the same directory has
already replaced it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* mount: remove the control socket before a failed mount exits

glog.Fatalf and glog.Exitf call os.Exit, which skips the deferred removal,
so a mount that failed after creating its socket (mount point not
available, fusermount failure) still left it behind.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* mount: register the socket cleanup hook before mount setup

A SIGTERM that arrived while the mount was still setting up, for example
waiting on the filer to create the mount root, exited through grace without
running the deferred cleanup. Registering the hook right after the listener
is created makes it run last, after the unmount and filesystem hooks.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-11 10:11:58 -08:00
Chris LuandClaude Opus 5.5 7f18576798 filer/etcd: serve volume.fsck from one etcd revision (#12555)
* filer/etcd: scan entries from one revision

volume.fsck walked etcd one directory at a time, so a file moved from a
directory not yet listed into one already listed was seen under neither
name and its chunks were reported as orphans. The etcd store now
implements filer.EntryScanner: every page is read at the revision of the
first read, and a compacted revision fails the scan instead of falling
back to the walk.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* filer/etcd: read hard links at the scan revision

Hard-link records resolved during a scan were read at the current
revision, so a link updated meanwhile mixed two instants. Reads made with
the scan's context now use its revision.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* shell: list etcd among the stores volume.fsck reads from a snapshot

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* filer/etcd: skip hard-link records in a scan

Hard-link records live beside the entries under their random ids, so a
root scan could read one as an entry with a path made of random bytes,
which the traversal stream cannot send. Skip hard-link ids and keys whose
directory is not valid UTF-8.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* filer/etcd: recognize a hard-link record by the id it stores

A hard-link id can be valid UTF-8, so the key shape alone let some
records through a root scan as phantom files. The record's value is the
shared entry, whose HardLinkId is the record's own key; no entry key
can equal its own hard-link id.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* filer/etcd: scan entries under an invalid-UTF-8 directory

The filer sanitizes names but stores directories as given, so a path
written over HTTP can leave entries the listing never reaches. Skipping
them let volume.fsck count their chunks as orphans. Send them under the
sanitized directory instead, the substitution the filer uses for names.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* filer/etcd: fail a scan whose sanitized path names another entry

An entry under an invalid-UTF-8 directory is reported under the sanitized
directory. When that path is also another stored entry at the scan revision,
or another sanitized entry in the same scan, acting on the reported path
(volume.fsck deleting it) would reach the other file, so the scan fails and
names the stored path instead.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-11 10:11:21 -08:00
e23a952878 ec: don't index ev.Shards[0] when the last shard was unmounted (#12538)
* ec: don't index ev.Shards[0] when the last shard was unmounted

* ec: skip a runtime emptied before the scrub merge reads its shards

FindAllEcVolumes releases each location lock before MergeEcRuntimes runs,
so an unmount can leave a merged runtime with no shards and the width
scan indexed Shards[-1].

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* ec: treat a scrub merge with no shard left as a vanished volume

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* ec: read Shards once in ShardSize so a concurrent unmount cannot empty it

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* ec: read each runtime's Shards once in the scrub merge

The vanished check and the slot construction read Shards separately, so
an unmount between them produced a non-nil view with no shards.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* ec: anchor the scrub merge on a generation that still holds shards

An unmount can empty the newest runtime while an older one keeps its
shards, which left a merged view anchored on the emptied generation with
no slots and the holder reported as skipped.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Mansur Akhmadi <10739460+mansur-akhmadi@users.noreply.github.com>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-11 09:57:39 -08:00
Chris LuandClaude Opus 5.5 641ab6204b shell: fs.mergeVolumes leaves in-progress multipart parts in place (#12556)
* shell: fs.mergeVolumes leaves in-progress multipart parts in place

CompleteMultipartUpload reads the part entries and later commits the
object with their fids, guarded only by the upload directory. A merge
that moved a part chunk in between deleted the source needle the new
object pointed at. Parts stay where they are until the upload completes
or is cleaned, and their chunks still count as referenced sources.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* shell: note in fs.mergeVolumes help that upload parts stay in place

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* shell: match upload parts when the buckets folder is the root

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* shell: count chunks behind a part's manifest as referenced sources

A skipped part whose chunks on a merge source sit behind a manifest was
not counted, so the merge printed no skip line and warned that nothing
references the source.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* shell: keep a part's direct source references when its manifest is unreadable

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* shell: count chunks resolved before an unreadable part manifest

ResolveChunkManifest returns the data chunks of earlier manifests along
with a later manifest's error; record them before returning so the
unreferenced-source warning does not fire for a source they reference.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* shell: test the partial manifest result without a timing delay

The unreadable manifest is now nested under a readable part manifest, so
the first level resolves fully before the failing lookup runs.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-11 09:55:03 -08:00
Chris LuandClaude Opus 5.5 e9606c8ecc filer.meta.backup: resume an interrupted metadata copy and add -entriesPerSecond (#12549)
* filer_pb: add a depth-first traversal that can resume after a path

TraverseDfs walks a tree in listing order and reports each finished
path, so a caller can save one path and continue from it later. Pages
are read whole before visiting, and the entries of a page are visited
by up to five workers.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* filer.meta.backup: resume an interrupted metadata copy

The walk saves the last finished path every few seconds, and a walk
that continues an unfinished one starts after it. A fresh walk clears
the saved path first, and a finished walk clears it with the walk
start.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* filer.meta.backup: add -entriesPerSecond to throttle the metadata copy

Each entry the full copy reads waits on a rate limiter, and pages are
listed only after the previous page is copied, so the flag also bounds
how fast the copy lists the source filer. 0 keeps it unlimited.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* filer_pb: report each file of a depth-first walk once all before it are done

A throttled walk only reported progress after a whole listing page, so a
slow copy could go many minutes without a resumable path. Visit entries in
runs that end at a directory and report each file as soon as every entry
before it is done.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-11 09:08:11 -08:00
Chris LuandClaude Opus 5.5 0ef0260844 volume: fail a vacuum step that finds the volume already compacting (#12553)
* volume: fail a vacuum step that finds a compaction already running

Compact, CompactByIndex and CommitCompact returned success when another
compaction held the volume. A second, uncoordinated vacuum (an expired
admin job still running, a shell volume.vacuum next to the worker) then
went on to commit or clean up the first run's .cpd/.cpx. Return a
"volume N is compacting" error instead, and refuse cleanup while a copy
is in flight, as the Rust volume server already did.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* volume: check the compaction files before a commit closes the volume

CommitCompact closed the needle map and data file and wrote the .cpc
marker before it found .cpd/.cpx missing. A duplicate commit, such as
the second of two overlapping vacuums, then left the volume registered
with no open files: it dropped out of the heartbeat and looked lost to
volume.list and volume.fix.replication until a restart. Check for the
files first so the refused commit leaves the volume as it was.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* volume: run makeupDiff before a commit closes the volume

CommitCompact closed the needle map and data file before makeupDiff, so
a makeupDiff failure (a compaction revision another commit moved, an I/O
error) returned with them closed: the volume stayed registered but
missing from the heartbeat until a restart. makeupDiff reads the live
files through its own handles while the commit holds the write lock, so
close them only once it has succeeded, as the Rust volume server does.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* volume: log a volume the heartbeat skips because its files are closed

A volume left without its needle map or data file vanished from the
heartbeat with no log line, so the master saw a replica disappear while
its files were still on disk. Warn on each skip, as the deleted-.dat
case next to it already does.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* volume: claim the compaction flag while cleanup removes the temp files

Cleanup only read the flag, so a compaction could start between the
check and the removal and lose its .cpd/.cpx. Claim it for the cleanup
instead, in Go and Rust.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-11 08:48:58 -08:00
Chris LuandClaude Opus 5.5 a4a68c885a filer.sync: move the watermark over jobs that finished out of order (#12554)
* filer.sync: move the watermark over jobs that finished ahead of an older one

The watermark only took the finishing job's timestamp when it was the
oldest in flight, so jobs that completed before an older one were never
recorded and a quiet stream kept the offset behind finished events.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* filer.sync: keep one finished timestamp per gap behind a running job

A long job kept every later completion in the done heap. Only the newest
one between two running or failed timestamps can become the watermark.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* filer.sync: test done-heap compaction across several running and failed jobs

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-11 08:47:30 -08:00
000baf8dfa s3: return an error for GETs canceled before the first byte (#12502)
* s3: answer GETs canceled before the first byte

A stream that fails before writing anything now always gets an S3 error
response, canceled errors included. The idle timeout and a shared lookup
canceled by another request both cancel the request context while the
client is still connected, and returning silently sent that client a
200 with Content-Length: 0. A client that has gone away never sees the
write.

Refs #12500

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3: mark inline GET write errors as written; bound the idle-cancel test

An inline object's write error comes after the status line, so the
caller must not write an S3 error on top of it. The idle-cancel test
caps the client wait and gives the response write its own deadline,
keeping only the read side on the 200 ms idle timeout.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3: test a GET whose request is canceled before the first byte

Drives GetObjectHandler through a chunk lookup that cancels the request
context, so a handler that returns before answering shows up as the empty
200 again.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3: check the cancel test reached the chunk lookup

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
2026-11-11 08:45:20 -08:00
Chris LuandClaude Opus 5.5 ea6e9d670a volume: forward no Content-MD5 for a JPEG the orientation fix rewrote (#12557)
* volume: forward no Content-MD5 for a JPEG the orientation fix rewrote

With -images.fix.orientation the primary stores and forwards the rotated
bytes, but sent replicas the client's digest of the original upload. The
replica checks Content-MD5 against what it receives and rejects the write,
so replicated JPEG uploads with a digest failed. The Rust volume server
already drops the digest once the body is rewritten.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* volume: test the forwarded Content-MD5 through PostHandler and a replica

The test now posts each upload to a primary whose fake master names a
real replica volume server, so it covers the JPEG gate, ReplicatedWrite
and the replica's own Content-MD5 check, and reads the stored bytes back.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* volume: fail the replica forwarding test instead of hanging when nothing is forwarded

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* volume: compare the primary and replica bytes in the replica forwarding test

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-11 08:44:06 -08:00
1202c5ba08 filer.copy: compute chunk md5 so files get a real ETag (#12497)
* filer.copy: compute chunk md5 so files get a real ETag

Chunks uploaded by filer.copy had no ETag, so every multi-chunk file got
the md5 of the empty string plus -N. An overwritten file kept its old ETag
and a revalidating client got 304 for changed content.

* filer.copy: store chunks in offset order

Chunks are collected in upload completion order, and the entry ETag
hashes the chunk ETags in stored order. Identical files copied with
concurrent chunk uploads got different ETags, and a file with swapped
equal-sized chunks could keep the old one.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* filer.copy: test chunk order and chunk md5 on chunked uploads

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-11 08:43:23 -08:00
b9866bae4f weed/shell: support -noLock flag in volumeServer.evacuate (#12501) (#12516)
* weed/shell: support -noLock flag in volumeServer.evacuate (#12501)

* shell: return flag parse errors from volumeServer.evacuate

A typo such as -noLock=invalid printed the parse error but the command
still succeeded, so a piped script treated the evacuation as done.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* shell: volumeServer.evacuate -h prints usage without failing

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-10 21:25:42 -08:00
09a63031c1 helm: set replication and disk on buckets created by the chart (#12543)
* helm: set replication and disk on buckets created by the chart

* helm: test bucket replication and disk without ttl

* helm: reject unquoted bucket replication values

* helm: treat a null bucket replication as unset

A blank replication key in a values file failed the render with the
quoted-string error, while blank ttl and disk are skipped.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* helm: require a three-digit bucket replication

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* helm: note that removing bucket settings keeps the stored rule

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* helm: reject bucket replication above 255

fs.configure rejects an xyz value above 255, so the hook Job failed after
the release rendered.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-10 21:24:55 -08:00
f068fe7f09 fix(shell): fs.log.purge purges every filer, not just one (#12524)
* fix(shell): fs.log.purge purges every filer, not just one

Each filer keeps its own metadata logs in its own store, so purging a
single filer leaves the chunks referenced by the others and the volume
vacuum cannot reclaim anything.

The shell picked that single filer at random (commands.go:208) when no
-filer flag was given, and the admin script worker pinned it to
filers[0] (admin_script_handler.go:574), so the same command purged a
different filer on every run and never the whole cluster.

fs.log.purge now resolves the filer list from the master and purges each
of them, continuing when one is unreachable and reporting the failures
at the end. -filer=<addr>,... restricts the purge to a subset.

Also stops mutating the shared command environment per filer, so a
long-lived shell keeps its original filer.

* fix(shell): bound filer discovery and cover the cluster path

Address review feedback:

- resolveLogPurgeFilers waited on the master with context.Background(): a
  shell whose master connection dropped would hang before purging
  anything, where it used to purge the connected filer. Bound the master
  round trip with a timeout and fall back to the connected filer when the
  master cannot be reached or lists no filer.
- the tests only exercised the single-filer fallback. Rename that test to
  what it actually covers and add one asserting every filer of the list is
  attempted and the shared command environment keeps its original filer.

* shell: reject an fs.log.purge -filer list with no addresses

A comma or whitespace only -filer value fell through to master
discovery and purged every filer instead of the intended subset.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* shell: fall back to the shell's filer without a second master lookup

When the master listed no filer, FilerAddress() ran filer discovery again
and could wait out the shell's ConnectTimeout before failing. Also cover
master discovery with the fake discovery master.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* shell: test the fs.log.purge filer loop against a fake filer

The previous test looped over the addresses itself, so it passed no
matter what Do did.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Martin Morez <martin@morez.org>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-10 21:24:04 -08:00
Chris LuandClaude Opus 4.8 fb5ffb815f filer: validate bucket names before the SQL DDL hooks (#12547)
* filer: validate bucket names before the SQL DDL hooks

The SQL store builds CREATE TABLE and DROP TABLE statements by formatting
the bucket name into the query. The DML paths already reject names that
fail VerifyS3BucketName, but the bucket-creation and bucket-deletion
hooks ran the DDL without that check, so a name reaching them through the
filer or WebDAV directory path (which skips the check for dot-prefixed
directories) could inject stacked statements. Apply the same validation
the DML paths use.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* filer: check that a valid bucket runs both DDL statements

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
2026-11-10 21:22:31 -08:00
Chris LuandClaude Opus 5.5 d350a0ad03 weed-db: end startup when password authentication fails (#12550)
* weed-db: end startup when password authentication fails

The failure branches returned the result of writing the error response,
which is nil on success, so the handshake carried on to ReadyForQuery.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* weed-db: test startup with wrong and unknown credentials

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* weed-db: check the auth error code and fail on stray read errors in the startup test

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-10 21:21:05 -08:00
Chris LuandClaude Opus 5.5 f3150c3495 admin: escape file properties before rendering the modal (#12548)
* admin: escape file properties before rendering the modal

The Properties modal concatenated the entry name, path, MIME type and
extended attributes into HTML, so object metadata written over S3 could
inject markup into the admin page.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* admin: keep every property key when escaping

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-10 21:19:53 -08:00
Chris LuandClaude Opus 5.5 8d96985131 security: scoped filer tokens are not admin credentials (#12551)
* security: do not accept a path- or method-scoped filer token as admin

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* security: test that scoped filer tokens fail the admin token check

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-10 21:18:59 -08:00
2143761319 fix(s3tables): let a table renamed across namespaces keep committing (#12515)
* s3tables: let a table renamed across namespaces keep committing

* s3tables: keep the same-directory exception off a directory another table shares

A record written before metadata locations were confined to their namespace
(6f93f874) and claimed (#12473) can keep its files in another table's
directory, in another namespace. The exception that lets a table renamed
across namespaces keep committing to the directory it left behind accepted
such a record's commits too, where master refused them.

The namespace check now runs first, as on master. Only a location it refuses
is matched against the record's current directory, and the exception holds
only when no other record keeps its files there. Where other records' entries
sit is ignored, so a table created under the renamed table's old name, whose
files go to a directory of its own, does not take the exception away. The
scan runs only on that path, so commits inside the namespace cost what they
did.

Co-Authored-By: Claude <noreply@anthropic.com>

* s3tables: seed the table at the renamed table's old name

A rename may keep the old name taken, so creating a table there can be
refused; seeding the record keeps the test on the update path.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude <noreply@anthropic.com>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
2026-11-10 21:15:48 -08:00
Tsic 518009dcd4 filer: apply filerGroup prefix to bucket default collection (#12499)
filer: prefix bucket default collection, assign copy collection explicitly

S3 server-side copy paths (CopyObject/UploadPartCopy) assign volumes with an empty collection and relied on the filer's bucket fallback. The fallback used the raw bucket name, while PutObject uses getCollectionName(bucket) = <filerGroup>_<bucket>, so copies landed in a second, unprefixed collection.

Prefix the filer's bucket default with the filer group, and have the S3 copy path pass that collection explicitly, like PutObject. Direct filer writes keep path-rule precedence, so this only aligns S3 copy with S3 put.

Adds regression tests for the fallback, explicit collection precedence, and the S3 copy collection resolver.
2026-11-10 21:12:49 -08:00
Ryan Leung 7ad8419815 shell: preserve completed uploads for bucket-root objects (#12545)
Signed-off-by: rleungx <rleungx@gmail.com>
2026-11-10 19:39:10 -08:00
github-actions[bot] 9799b82edf docs: regenerate star history chart 2026-11-11 01:16:36 +00:00
Mansur AkhmadiandMansur Akhmadi 9886ac0bcf topology: take dn.Lock when a heartbeat links a new disk type (#12535)
Co-authored-by: Mansur Akhmadi <10739460+mansur-akhmadi@users.noreply.github.com>
2026-11-10 12:17:21 -08:00
Mansur AkhmadiandMansur Akhmadi 9f4b652522 master: make admin lock lease and release atomic (#12532)
Co-authored-by: Mansur Akhmadi <10739460+mansur-akhmadi@users.noreply.github.com>
2026-11-10 12:16:23 -08:00
Saarthak Gupta 017e42633f shell: return flag parse errors from s3.bucket.create (#12544) 2026-11-10 12:15:01 -08:00
Mohd Quamar TyagiandChris Lu 5863e32318 weed/s3api: avoid truncated marker rescan in ListObjects (#12540) (#12541)
* weed/s3api: avoid truncated marker rescan in ListObjects (#12540)

Signed-off-by: Tyagiquamar <mohdquamartyagi@gmail.com>

* fix(s3): validate marker directory candidates

Respect prefix filters and internal-directory exclusions when recovering marker candidates. Restrict lookups to direct children, propagate read failures other than not-found, and deduplicate directory marker echoes.

Add regression coverage and model promoted objects consistently in the XML-unsafe prefix fixture.

---------

Signed-off-by: Tyagiquamar <mohdquamartyagi@gmail.com>
Co-authored-by: Chris Lu <chris.lu@gmail.com>
2026-11-10 09:58:45 -08:00
Chris Lu d485d2c9cd filer.sync: shorten retries for confirmed missing chunks (#12546)
* filer.sync: shorten retries for confirmed missing chunks

* filer.sync: clarify missing chunk grace period names
2026-11-10 09:56:17 -08:00
Chris Lu 6c473433c5 vector: bound request buffering before authentication and document installation (#12537)
* s3vectors: bound body buffering before authentication

* docs: include vector engine installation and source builds

* s3vectors: prevent MIME sniffing of JSON error responses

* docs: load vector Cargo configuration when building from source

* docs: note the native protoc requirement for Windows vector builds

* docs: make vector source checkout and working directory explicit

* ci: handle deb822 sources during Ubuntu mirror failover

* ci: bound Ubuntu mirror attempts and reject partial index updates
2026-11-10 04:21:56 -08:00
Chris Lu 8236c8d136 vector: integrate engine health, admin management, and local deployment (#12517)
* vector: expose request metrics and registration-aware health endpoints

* vector: report immutable read cache and storage fetch metrics

* docs: describe vector engine health, metrics, and maintenance

* vector: count canceled requests when their handler is dropped

* vector: account for uncached large immutable storage reads

* vector: reliably release the stalled credential reader in tests

* vector: observe large reads as their streamed body is consumed

* vector: bound heartbeats on established master registration streams

* test: exercise hardened vector maintenance on Go and Rust volumes

* vector: register engines in their configured filer group

* vector: make retried index creation preserve compatible datasets

* vector: allow registration to advertise a separate address

* test: release stalled credential readers without blocking FIFO cleanup

* test: allow hardened vector coverage in a named filer group

* vector: discover engines in the same filer group as their gateway

* vector: fall back when the advertised address is empty

* admin: expose vector readiness, maintenance, and management tools (#12520)

* s3vectors: share vector discovery with trusted management clients

* admin: show vector engine readiness and recent maintenance activity

* admin: suggest S3 Vectors IAM actions from the supported operations

* admin: manage vector indexes, policies, tags, encryption, and query data

* admin: validate SSE-S3 support before creating encrypted vector buckets

* admin: discover vector engines in their registration group

* admin: probe vector engines concurrently with independent time budgets

* admin: verify encryption capability on the remote S3 gateway

* admin: defer discovery until after the HTTP server can start

* admin: distinguish unknown vector health services from failed engines

* admin: count workers that can execute vector index maintenance

* admin: allow read-only users to query vectors and inspect tags

* vector: scope gateway and admin discovery to the configured filer group

* admin: refresh policy state after mutations and serialize policy controls

* vector: drain active RPCs before closing replaced discovery connections

* vector: respect request cancellation while waiting for discovery

* vector: prefer another registered engine after a transport failure

* admin: check all masters before reporting no registered vector engines

* iam: honor attached vector policies without broadening legacy grants

* vector: allow short-lived discovery clients to close their connections

* vector: refresh cached master addresses for each discovery lookup

* iam: enforce vector bucket tag conditions in attached policies

* iam: authorize vector bucket creation with the requested tags

* test: exercise vector IAM grants and tag denies on Go and Rust volumes

* admin: reject authenticated vector mutations without an admin role

* test: run vector IAM assertions before the shared engine is stopped

* vector: add native mini startup, shell tools, and a runnable Compose example (#12522)

* s3vectors: share vector discovery with trusted management clients

* admin: show vector engine readiness and recent maintenance activity

* admin: suggest S3 Vectors IAM actions from the supported operations

* admin: manage vector indexes, policies, tags, encryption, and query data

* admin: validate SSE-S3 support before creating encrypted vector buckets

* admin: discover vector engines in their registration group

* admin: probe vector engines concurrently with independent time budgets

* admin: verify encryption capability on the remote S3 gateway

* admin: defer discovery until after the HTTP server can start

* admin: distinguish unknown vector health services from failed engines

* admin: count workers that can execute vector index maintenance

* admin: allow read-only users to query vectors and inspect tags

* vector: scope gateway and admin discovery to the configured filer group

* admin: refresh policy state after mutations and serialize policy controls

* vector: drain active RPCs before closing replaced discovery connections

* vector: respect request cancellation while waiting for discovery

* vector: prefer another registered engine after a transport failure

* admin: check all masters before reporting no registered vector engines

* iam: honor attached vector policies without broadening legacy grants

* vector: allow short-lived discovery clients to close their connections

* vector: refresh cached master addresses for each discovery lookup

* iam: enforce vector bucket tag conditions in attached policies

* iam: authorize vector bucket creation with the requested tags

* test: exercise vector IAM grants and tag denies on Go and Rust volumes

* admin: reject authenticated vector mutations without an admin role

* test: run vector IAM assertions before the shared engine is stopped

* mini: optionally start and supervise the vector engine

* mini: bootstrap vector buckets with existing Lance catalog helpers

* shell: manage vector buckets, indexes, and vector data

* docs: provide a runnable vector and maintenance worker Compose stack

* shell: scope vector discovery to the session and close its connections

* mini: honor the bind address for vector and metrics listeners

* mini: reserve vector ports before assigning gRPC and compatibility listeners

* mini: wait for vector storage setup and master registration

* mini: create requested vector buckets without requiring the Lance HTTP listener

* shell: refresh master addresses during vector discovery

* mini: resolve sibling vector executables with platform extensions

* vector: preserve uncertain creates and fix mini listener allocation (#12525)

* vector: retain catalog records when index creation has an unknown outcome

* mini: keep chosen listener ports reserved throughout allocation

* mini: resolve hostname binds before starting vector listeners

* vector: fix integration test imports and recover interrupted index creation (#12533)

* shell: register vector operations from the command package

* s3vectors: resume interrupted index initialization safely

* mini: exclude disabled WebDAV from listener allocation

* s3vectors: retain uncertain creates across refused retries

* s3vectors: claim pending indexes before retrying initialization

* s3vectors: compare resolved distance metrics during recovery

* s3vectors: preserve metadata keys when retrying initialization

* s3vectors: verify ownership before initializing new catalog records
2026-11-10 04:21:18 -08:00
github-actions[bot] 3f5b07475d docs: regenerate star history chart 2026-11-10 00:45:43 +00:00
Chris Lu 5abb18e678 helm: manage vector engines and provision vector buckets (#12518)
* helm: deploy vector engines with TLS, probes, and monitoring

* helm: provision vector buckets through the S3 Vectors API

* helm: bracket IPv6 pod addresses for vector listeners

* helm: discover vector engines through the all-in-one master

* helm: accept vector component NetworkPolicy overrides

* helm: bind vector metrics to the pod address on both IP families

* helm: provision vector buckets through embedded and all-in-one gateways

* docs: use the full Helm name for external vector certificates

* helm: render vector instance labels as strings

* helm: allow vector bucket hooks to use restricted security contexts

* helm: honor vector security context enable switches

* helm: prevent duplicate vector monitoring labels

* helm: reject metrics flags with the default pre-metrics vector image

* test: skip vector chart compatibility checks when Helm is unavailable
2026-11-09 16:34:57 -08:00
Chris Lu c1afb1b1cb ci: finish native vector checks and fix Windows installer tests (#12526)
* test: pass installer scripts through stdin on Windows

* ci: allow cold Intel macOS vector builds to finish

* ci: retain completed Rust builds when later platform checks fail
2026-11-09 16:33:50 -08:00
Chris Lu 89d3016d5e vector: publish native macOS and Windows engines and installer support (#12523)
* ci: build, test, and publish vector engines for macOS and Windows

* installer: install native vector archives on macOS and Windows

* ci: run desktop vector checks for shared Rust and protobuf changes

* ci: publish vector releases from a separate job with write permissions

* ci: build tagged vector releases without restoring caches

* ci: propagate Windows bash path conversion failures
2026-11-09 16:14:42 -08:00
Chris Lu ac2857911d grafana: add vector API, engine, cache, and maintenance views (#12519)
* grafana: add vector API, engine, cache, and maintenance panels

* helm: include vector panels with Kubernetes metric selectors

* grafana: preserve cluster and namespace identity when matching engine health

* grafana: retain engine identity when scrapes fail

* grafana: show zero cache hits before the first hit

* grafana: keep cluster and namespace identity in vector cache ratios
2026-11-09 16:00:02 -08:00
dependabot[bot] d9d8942caf build(deps): bump docker/login-action from 4.7.0 to 4.8.0 (#12510)
Bumps [docker/login-action](https://github.com/docker/login-action) from 4.7.0 to 4.8.0.
- [Release notes](https://github.com/docker/login-action/releases)
- [Commits](https://github.com/docker/login-action/compare/v4.7.0...v4.8.0)

---
updated-dependencies:
- dependency-name: docker/login-action
  dependency-version: 4.8.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-11-09 11:59:14 -08:00
dependabot[bot] f39b1abd03 build(deps): bump github.com/shirou/gopsutil/v4 from 4.26.7 to 4.26.10 (#12507)
Bumps [github.com/shirou/gopsutil/v4](https://github.com/shirou/gopsutil) from 4.26.7 to 4.26.10.
- [Release notes](https://github.com/shirou/gopsutil/releases)
- [Commits](https://github.com/shirou/gopsutil/compare/v4.26.7...v4.26.10)

---
updated-dependencies:
- dependency-name: github.com/shirou/gopsutil/v4
  dependency-version: 4.26.10
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-11-09 11:59:07 -08:00
dependabot[bot] 2373207b6e build(deps): bump sigstore/cosign-installer from 4.1.2 to 4.1.3 in /.github/actions/sign-image (#12512)
build(deps): bump sigstore/cosign-installer

Bumps [sigstore/cosign-installer](https://github.com/sigstore/cosign-installer) from 4.1.2 to 4.1.3.
- [Release notes](https://github.com/sigstore/cosign-installer/releases)
- [Commits](https://github.com/sigstore/cosign-installer/compare/v4.1.2...v4.1.3)

---
updated-dependencies:
- dependency-name: sigstore/cosign-installer
  dependency-version: 4.1.3
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-11-09 11:58:55 -08:00
dependabot[bot] 82cd43a884 build(deps): bump helm/kind-action from 1.15.1 to 1.15.2 (#12511)
Bumps [helm/kind-action](https://github.com/helm/kind-action) from 1.15.1 to 1.15.2.
- [Release notes](https://github.com/helm/kind-action/releases)
- [Commits](https://github.com/helm/kind-action/compare/v1.15.1...v1.15.2)

---
updated-dependencies:
- dependency-name: helm/kind-action
  dependency-version: 1.15.2
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-11-09 11:58:40 -08:00
dependabot[bot] d1d081fcf2 build(deps): bump github/codeql-action from 4.39.1 to 4.39.3 (#12509)
Bumps [github/codeql-action](https://github.com/github/codeql-action) from 4.39.1 to 4.39.3.
- [Release notes](https://github.com/github/codeql-action/releases)
- [Changelog](https://github.com/github/codeql-action/blob/main/CHANGELOG.md)
- [Commits](https://github.com/github/codeql-action/compare/v4.39.1...v4.39.3)

---
updated-dependencies:
- dependency-name: github/codeql-action
  dependency-version: 4.39.3
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-11-09 11:58:24 -08:00
Chris Lu 84cd3c621b s3vectors: integrate bucket metrics, audit events, and request limits (#12514)
* s3vectors: attribute request and traffic metrics to authorized buckets

* s3vectors: emit resource and identity audit events

* s3vectors: apply existing S3 request limits and preserve JSON errors

* s3vectors: select limit resources by the dispatched operation

* s3vectors: count decoded chunked request bodies against byte limits
2026-11-09 11:37:06 -08:00
Chris Lu 620a1c3f39 docs: showcase vector buckets and cluster features (#12513)
* docs: showcase vector buckets and cluster features

* docs: include built-in Iceberg catalog in table bucket overview

* docs: keep feature diagram at cluster level
2026-11-09 08:52:12 -08:00
dependabot[bot] 643784df0d build(deps): bump github.com/aws/aws-sdk-go-v2 from 1.47.5 to 1.47.7 (#12506)
Bumps [github.com/aws/aws-sdk-go-v2](https://github.com/aws/aws-sdk-go-v2) from 1.47.5 to 1.47.7.
- [Release notes](https://github.com/aws/aws-sdk-go-v2/releases)
- [Commits](https://github.com/aws/aws-sdk-go-v2/compare/v1.47.5...v1.47.7)

---
updated-dependencies:
- dependency-name: github.com/aws/aws-sdk-go-v2
  dependency-version: 1.47.7
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-11-09 08:29:08 -08:00
dependabot[bot] 9bb33d0e36 build(deps): bump github.com/Azure/azure-sdk-for-go/sdk/azcore from 1.23.3 to 1.24.0 (#12505)
build(deps): bump github.com/Azure/azure-sdk-for-go/sdk/azcore

Bumps [github.com/Azure/azure-sdk-for-go/sdk/azcore](https://github.com/Azure/azure-sdk-for-go) from 1.23.3 to 1.24.0.
- [Release notes](https://github.com/Azure/azure-sdk-for-go/releases)
- [Commits](https://github.com/Azure/azure-sdk-for-go/compare/sdk/azcore/v1.23.3...sdk/azcore/v1.24.0)

---
updated-dependencies:
- dependency-name: github.com/Azure/azure-sdk-for-go/sdk/azcore
  dependency-version: 1.24.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-11-09 08:28:40 -08:00
dependabot[bot] a1ba470b32 build(deps): bump github.com/testcontainers/testcontainers-go from 0.44.0 to 0.45.0 (#12504)
build(deps): bump github.com/testcontainers/testcontainers-go

Bumps [github.com/testcontainers/testcontainers-go](https://github.com/testcontainers/testcontainers-go) from 0.44.0 to 0.45.0.
- [Release notes](https://github.com/testcontainers/testcontainers-go/releases)
- [Commits](https://github.com/testcontainers/testcontainers-go/compare/v0.44.0...v0.45.0)

---
updated-dependencies:
- dependency-name: github.com/testcontainers/testcontainers-go
  dependency-version: 0.45.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-11-09 08:28:27 -08:00
dependabot[bot] 5b87b7bc95 build(deps): bump github.com/prometheus/procfs from 0.22.0 to 0.23.0 (#12503)
Bumps [github.com/prometheus/procfs](https://github.com/prometheus/procfs) from 0.22.0 to 0.23.0.
- [Release notes](https://github.com/prometheus/procfs/releases)
- [Commits](https://github.com/prometheus/procfs/compare/v0.22.0...v0.23.0)

---
updated-dependencies:
- dependency-name: github.com/prometheus/procfs
  dependency-version: 0.23.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-11-09 08:28:15 -08:00
github-actions[bot] 7506e5f91c docs: regenerate star history chart 2026-11-09 00:35:41 +00:00
Chris LuandClaude Opus 5.5 5c910ad546 topology: list collections without nesting the collection map read lock (#12495)
* topology: list collections without nesting the collection map read lock

Items() already takes the read lock and copies the values. Holding it
around the call made ListCollections take the read lock twice, so a
DeleteCollection waiting for the write lock in between blocked the
second read lock and wedged the master.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* topology: test ListCollections against a concurrent collection delete

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-08 15:05:41 -08:00
github-actions[bot] 4ab66fb76d 4.53 2026-11-08 18:10:33 +00:00
Chris LuandClaude Opus 5.5 76fddab5ef master: keep a re-registered filer in the lock ring when its old stream ends (#12494)
* master: move the KeepConnected stream cleanup into removeClusterNode

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* master: drop a filer from the lock ring only with its last registration

A filer's replacement can register under the same address before the
master ends the old stream. The cluster list counts both registrations,
but the old stream's cleanup removed the address from the lock ring
outright, leaving the live filer out of the ring until it re-registered.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* master: order a stream's cluster list and lock ring changes

The cleanup of an ending stream could see its address removed from the
cluster list, then a new registration of that address add it to the list
and the ring, and only then remove it from the ring.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* master: take a leaving filer's reconnect out of the lock ring

A filer that reconnects while leaving registers with LeaveLockRing set.
Its broken stream can still be open on the master, and since that
stream's end no longer drops an address that is still registered, the
reconnect removes the address from the ring itself.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-08 09:55:32 -08:00
Chris Lu 35e98177d9 shell: test collection.list keeps live statistics next to offline names (#12492) 2026-11-08 09:49:32 -08:00
Ping Qiuandcodex03 0705e20fe9 shell: include collection names when topology statistics are unavailable (#12490)
fix(shell): include collections without live topology in list

Co-authored-by: codex03 <codex03@local.invalid>
2026-11-08 09:32:47 -08:00
Chris LuandClaude Opus 5.5 c147b775f9 helm: say what the Lance worker needs from security.toml (#12489)
* helm: say what the Lance worker needs from security.toml

Since #12470 the Lance namespace authenticates weed-worker by a token
derived from jwt.filer_signing.key, which the chart renders only with
securityConfig.jwtSigning.filerWrite. Without it, a gateway with
identities lists the worker no tables, and the lance_* jobs silently
find nothing to do.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* helm: warn that filerWrite also requires signed filer HTTP writes

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-08 09:19:51 -08:00
Chris LuandClaude Opus 5.5 4e12c13416 Volume encryption follows the filer: weed s3, weed-vector, weed-worker, weed-vector-cli (#12487)
* s3: encrypt volume data when the filer is configured to

weed s3 read the filer's configuration at startup (buckets dir, filer
group, masters, metrics, maxMB) but not its -encryptVolumeData, so a
cluster that turned encryption on at the filer still had S3 uploads
stored plain, while uploads through the filer itself were encrypted.
The gateway now encrypts when either its own -encryptVolumeData or the
filer's is set; weed-vector, weed-worker and weed-vector-cli already
follow the filer's.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* vector: weed-vector, weed-worker and the CLI take volume encryption from the filer

They already encrypted chunks for the volume servers when the filer's
configuration said so, or when their own --encryptVolumeData did. With
weed s3 following the filer too, the filer's -encryptVolumeData is the
one switch for every writer, so the three flags are removed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* S3 Vectors tests: turn volume encryption on at the filer

VECTOR_ENCRYPT_VOLUMES=1 set -encryptVolumeData on the gateway, the
engine and the worker. It now sets it on the filer alone, so the leg
checks that each of them follows the filer: the gateway's stored chunks
carry a cipher key (with a gateway that ignores the filer, the leg fails
on "the gateway stored a chunk without a cipher key"), and so do the
engine's.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* weed-worker README: volume encryption follows the filer

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* s3: help text says the filer's -encryptVolumeData turns S3 encryption on

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* S3 Vectors tests: plain legs check the engine stored no cipher key

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-08 09:17:30 -08:00
Chris LuandClaude Opus 5.5 87627c2d72 S3 Vectors: help text for weed-vector, weed-vector-cli and weed-worker (#12486)
vector: the engine's, CLI's and worker's help say what they do now

weed-vector's --masters and --filer-group said "reads through" and
--security-toml named only gRPC TLS, though the engine keeps tables on the
filers and volume servers and takes https.client, the read JWT key and the
SSE-S3 keys from it too; weed-vector-cli called itself a "query spike";
weed-worker's --filer-signing-key spoke of write credentials the
namespace no longer vends.


Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-08 05:55:40 -08:00
Chris LuandClaude Opus 5.5 af299655e7 S3 Vectors: weed-vector drops the ignored --namespace and --filer-signing-key (#12485)
vector: weed-vector drops the ignored --namespace and --filer-signing-key

Since #12468 the gateway resolves every table it sends, so the engine has
no catalog to name or authenticate to; the two flags were kept only so an
older command line still started, with a warning. The direct-storage
engine is new, and release 4.52's command lines need changing anyway
(--storage-option is refused), so they are removed.


Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-08 05:55:30 -08:00
Chris LuandClaude Opus 5.5 cd616a48a8 S3 Vectors: refuse an SSE-S3 index when the engine has no key for it (#12484)
vector: refuse an SSE-S3 index when the engine has no key for it

The engine writes and reads an SSE-S3 bucket's files itself, so without
s3.sse.kek or s3.sse.key it failed inside Lance and the client saw only
InternalFailure. weed mini makes this easy to hit: it generates its key
for its own S3 server only. A table the gateway marks sse_s3 is now
refused up front with FailedPrecondition, which the gateway answers as
ValidationException with the reason.


Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-08 05:13:19 -08:00
Mansur AkhmadiandMansur Akhmadi 10ea662522 s3: don't report a version delete as done when the filer lookup failed (#12482)
* s3: don't report a version delete as done when the filer lookup failed

deleteSpecificObjectVersion treated any getEntry error as "entry is gone":

- the null-version precheck returned success without deleting anything,
- the recheck after a failed rm returned success while the entry may
  still be there,
- a failed .versions lookup was read as "not the latest version", so the
  latest blob was removed without repointing the pointer first.

Only filer_pb.ErrNotFound now counts as a miss. Other errors fail the
delete, so the S3 client or the lifecycle worker retries it. The lookups
go through retryFilerOp, like the other .versions reads in this file, so
a single transient filer error doesn't turn into a 500.

* s3: include recheck error when a version delete fails

* s3: recheck a failed version delete once instead of retrying

---------

Co-authored-by: Mansur Akhmadi <10739460+mansur-akhmadi@users.noreply.github.com>
2026-11-08 05:06:52 -08:00
f99cc6ca94 volume.fsck: batch and deduplicate filer entry deletions (#12480)
* volume.fsck: batch and deduplicate filer entry deletions

In volume.fsck with -findMissingChunksInFiler and -reallyDeleteFilerEntries, deleting orphaned filer entries was slow and CPU intensive due to tiny worker channel buffers (128), lack of batch processing, redundant HTTP DELETE requests for multi-chunk files, and lock contention on parent directory tracking.

Optimize by:
- Adding startPurgeBatchWorkers with 4096-buffered channels to prevent producer stalls and batch opportunistic deletes up to 64 items.
- Deduplicating missing chunks per file path during volume checks so that multi-chunk files are queued once.
- Processing batches with in-batch path deduplication and aggregating parent directories before updating purgedDirs in a single lock acquisition.
- Buffering temporary fid file writes (64KB) during the traversal to avoid millions of unbuffered syscalls.

* volume.fsck: report a failed write of the filer fid files

The fid files are written through a buffer now, so the last part of
each one reaches the disk only in the deferred Flush. Its error was
dropped: a full temp disk left a short fid file and the collection
still succeeded, and the volume check would then take the chunks
missing from it for orphans.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* volume.fsck: retry a batched path whose earlier delete failed

A path was marked done before its delete ran, so a second copy of it in
the same batch was skipped after a transient failure and the entry
stayed. Only skip paths already deleted.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* volume.fsck: keep reporting a failed DELETE response read

The non-verbose path now discards the body without checking the copy,
so a connection cut after the headers went unreported.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* volume.fsck: keep one purge worker starter and one delete function

startPurgeWorkers and the single-path httpDelete were left only as
wrappers the tests called. Give their names to the batch versions and
adapt the tests.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-08 05:03:52 -08:00
Copilotandchrislusf 73fa85f54f Retry transient failures in star history generation (#12479)
* Initial plan

* Retry transient star history API failures

Co-authored-by: chrislusf <1543151+chrislusf@users.noreply.github.com>

---------

Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Co-authored-by: chrislusf <1543151+chrislusf@users.noreply.github.com>
2026-11-07 18:12:34 -08:00
Chris LuandClaude Opus 5.5 4f24b756ba s3tables, iceberg: give tables created without a location their own directory (#12475)
* s3tables: give a table created without a location its own directory

An Iceberg table created without a location kept its files under its name
path, so a rename left them where the next table of that name would land.
Such a table now gets s3://<bucket>/<namespace>/<name>-<uuid>, stored as the
record's warehouseLocation and returned by GetTable; its first commit writes
there without moving the root. Records without one keep resolving to their
name path, so existing tables need no rewrite. Lance datasets stay at their
name path, where Lance clients and the vector engine look for them.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* iceberg: create tables and views in a directory of their own

A table or view created without a location took its name path, or a
unique one only when that path was already taken. It now always gets
<name>-<uuid>, so a later rename never leaves its files under a name
another table can take.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* iceberg: load and commit a table at its warehouse location

A table created through S3 Tables without a location has no metadata until
its first commit; loading or committing it now uses the directory recorded
for it, and only a record without one falls back to its name path.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* iceberg: fail a stage-create whose marker is not saved

The marker is how the commit finds the location the stage-create handed
out. Without it the commit fell back to the name path while the client's
files sat under the returned location.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test: expect tables created without a location in their own directory

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test: rename a table that keeps its files elsewhere before reusing its name

Enterprise keeps the storage marker on a renamed table's files, so the name
of a table whose files sit under its name path stays taken there.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3tables: remove the table directory when a create backs out

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* iceberg: answer a create that lost the race with the registered table

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* iceberg: answer a view create that lost the race with the registered view

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* iceberg: fail a staged commit whose name another create registered

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* iceberg: remove the directory of a create whose name another table took

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* iceberg: keep the metadata of a staged commit whose name is taken

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test: write the lakekeeper table's data under its warehouse location

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* iceberg: commit a staged table at the location of its own stage

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* iceberg: never continue another stage when a commit's own marker is gone

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* iceberg: test filer honors exclusive creates and token-guarded deletes

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-07 16:37:31 -08:00
Chris LuandClaude Opus 5.5 55a4a5974c ec_balance: make the per-rack cap on global moves a worker setting (#12478)
* ec_balance: make the per-rack cap on global moves a worker setting

The global load-balancing phase planned at most 10 moves per rack per
detection run, hard-coded. On a cluster where one disk per machine holds
several times its siblings' shards, that is about 20 moves an hour against
tens of thousands needed, however many moves the worker can execute.

Add global_max_moves_per_rack to the plugin worker config. The default stays
10, so nothing changes unless it is raised. It is clamped to 1..5000 because
the planner reads 0 as unlimited and would plan a whole drain in one run.

Claude-Session: https://claude.ai/code/session_011UpvafcfUuC2mLxNAXLQJi

* ec_balance: clamp global_max_moves_per_rack before narrowing it to int

A value past the int32 range made ReadIntConfig return the default, so a
request asking for more than the maximum got 10 moves instead of 5000.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* ec_balance: test the configured global cap through Detection

The helper test still passed if Detection handed the planner a literal cap.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* ec_balance: plan no more global moves per rack than the detection keeps

Each global move scans the source node's volumes, and the detection keeps
at most max_results moves. With the cap at 5000, three skewed racks of
20,000 volumes planned 15,000 moves to keep 500, taking about 70 s of the
300 s detection budget; bounded by max_results it takes about 5 s.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* ec_balance: test the configured global cap with a result limit

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-07 16:01:12 -08:00
78a3bc9dc3 filer.remote.sync: -allMounts writes back every mounted directory from one process (#12476)
* filer.remote.sync: -allMounts writes back every mounted directory from one process

A filer with many remote mounts needs one filer.remote.sync process per
mount. -allMounts runs the existing -dir write-back once per directory in
the mount table, in one process: each with its own subscription, client id
(the filer keeps one subscription per client id), offset, processor and
retry loop. A supervisor reads the mount table every -mountsCheckInterval
(5s) and starts a write-back for each mounted directory that has none
running, 10 at a time, 500 ms apart. It never stops one because the live
table changed: a write-back ends itself where the -dir process exits, at the
first mount-table version it receives that ends its mount (#12472), and a
directory still mounted is started again from its saved offset, after a
pause that grows from 1s to 30s while it keeps ending within a minute. A
directory mounted while the process runs, without a saved offset, waits up
to 2 minutes for one (a mount followed by a pull sets it after the pull). A
write-back that ends at an unmount 5 times in a row while the table still
mounts it at the same location is logged as a warning.

To run under a supervisor, a follow:

- returns errMountGone at the end of its mount instead of exiting.
  admitUntilMountEnds takes the end action: the -dir process keeps draining
  and exiting as before; a -allMounts follow admits nothing more (events
  after the end can still arrive in the same batch or before the stream
  closes, including remote.unmount's purge and a remount), ends its stream,
  drains, and saves no offset.
- drains its admitted jobs before it returns, however it ended, so a
  successor never races a job still running for the same path.
- saves its offset only over the value it read at its start or last saved
  (offsetGuard), both at its checkpoints and at its end. Otherwise a
  write-back behind in the log can overwrite the offset remote.unmount
  writes after its purge (or a mount after its mapping write), and its
  successor replays the unmount and never writes the remounted directory
  back. Once it sees the offset moved it saves nothing more and keeps no
  resume position. A value whose write reported an error counts as its own
  when read back.
- reads the stored offset once, before the mount table, and uses it as both
  its start and its guard's value: read after the table, an unmount
  completing in between would start a follow past the unmount's table.
- keeps a resume position for its next attempt only with the mount location
  it was earned under and the offset stored at the time; it is dropped when
  the directory is mounted elsewhere or someone else moved the offset.
- turns a panic in a job into that job's error, so one directory's failure
  does not end every directory's write-back.

Fixes #12471

* filer.remote.sync -allMounts: retire a failed follow whose mount is gone

A follow that failed before its stream reached the unmount (or before it
started) was retried every 30s forever: DetectMountInfo reports the
directory not mounted, which is not errMountGone. No stream runs on the
retry path, so a directory the live table no longer maps is retired there
and started again by reconcile if it is mounted again.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* filer.remote.sync -allMounts: drop the restart pause of an unmounted directory

ended() left notBefore and ends behind for every directory ever mounted.
reconcile now drops them once the directory is neither mounted nor
followed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* filer.remote.sync: log a -dir drain held at the mount end

The -dir exit drained with processor.Drain, so a remote call hung during
an unmount or move held the process silently; the follow's own drain
already warns every 5 minutes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* filer.remote.sync: keep the resume position through a failed mount read

A transient DetectMountInfo error dropped the only position behind the
event a follow failed on. With -timeAgo and no checkpoint saved yet, the
retry then started at a later now minus timeAgo and skipped that event.
The next attempt that reads the table still checks the position against
the mount location and the stored offset.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* filer.remote.sync: judge a kept resume position by the directory's remote location

resumeFor compared the mount's raw location, so a -dir below a mount kept
its position when the same remote was mounted one level up (or down),
although its paths now map to other keys; a replay from that position
skips startAfterMountOffset and writes old events to the new keys. The
position now carries dirRemoteLocation and is checked against it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* filer.remote.sync -allMounts: warn about a replayed unmount only from an unmoved offset

The warning counted every end at an unmount while the directory stayed
mounted at one location, so write-backs that each saved progress, or
unmount and remount cycles between two reconciles, also tripped it, and
it told the operator to set the offset to now, skipping events. An end
saves no offset, so a run of ends is one replay only while the stored
offset stays put: a moved offset now starts a new run.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* filer.remote.sync -allMounts tests: wait for the supervisor before moving the frozen clock

A reconcile or end recorded after the clock moved would start its wait
or pause at the moved time, which the frozen clock never passes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* remote.mount: save the sync offset right after the mount mapping

A new mount had no saved offset, so filer.remote.sync -allMounts waited
the full allMountsOffsetWait before following it and then started from
the directory's creation time anyway. remote.mount and remote.mount.buckets
now save the time taken just before the mapping write, keeping an offset
remote.unmount saved for a remount.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* remote.mount: save the directory's creation time as the sync offset

The mount time skipped local files already in a -nonempty directory, which
a write-back with no offset replays from the directory's creation. Saving
that creation time keeps the start position and only ends the -allMounts
wait.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* remote.mount: warn instead of failing when the sync offset is not saved

The mount mapping is already saved by then, and a mount without an
offset still starts from the same creation time after the -allMounts
wait. Failing stopped remote.mount.buckets before the remaining buckets.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-07 15:43:44 -08:00
Chris LuandClaude Opus 5.5 554593796e ec_balance: drain a server holding far more shards than its siblings (#12477)
* ecbalancer: drain a full server through any target that can take a shard

The global phase paired the fullest server with the single emptiest one and
stopped for the whole rack when that pair had no legal move. Where every
volume already has a shard on each machine, no cross-machine move keeps the
spread, so on hosts running one volume server per disk the first server's
disk kept its pile while its siblings on the same machine stayed nearly
empty: the phase never planned a move.

Try sources fullest first and targets emptiest first and take the first pair
with a legal move, so such a server drains onto its own machine's emptier
servers. A same-machine move goes only to a server holding none of the
volume, so one disk never carries two of its shards. Per-machine counts are
indexed once per rack and pairs without a move are remembered until a move
can change them, so an uncapped drain stays fast.

Claude-Session: https://claude.ai/code/session_011UpvafcfUuC2mLxNAXLQJi

* ec_balance: keep global moves ahead of within-rack moves under the detection cap

Detection truncated the plan to the per-cycle job limit in planner order,
which puts the global moves last. A long within-rack backlog pushed all of
them past the limit, so even a planned global move never ran. Order dedup
and cross-rack first, then the capped global phase, then within-rack.

Claude-Session: https://claude.ai/code/session_011UpvafcfUuC2mLxNAXLQJi

* ec_balance: keep each volume's moves in plan order when promoting global moves

The admin scheduler runs only the first proposal per volume each cycle. A
global move promoted ahead of its volume's within-rack moves can take a shard
that one of those moves placed, so it ran while the shard was still on the
original server and failed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* ecbalancer: order global targets by fullness after taking the shard

The search stops at the first target that would end up fuller than the
source, which only holds when targets are ordered by that same value. With
unequal capacities an empty one-slot server sorted first and ended the search
before a larger server that could take the shard.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* ecbalancer: free the source disk slot on a global move

The global phase cleared the moved shard's bits but left the source disk's
free slots and shard count unchanged, unlike every other phase. Use
releaseShard so a later pick of that server sees the freed disk.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* ec_balance: test that a capped detection keeps a global move

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* ecbalancer: test the same-machine move skips a sibling holding the volume

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-07 12:55:34 -08:00
Nguyễn Lộc Phúc 195f8156de fix(iceberg): keep a losing table create from deleting the winner's metadata (#12474)
* iceberg: keep a losing table create from deleting the winner's metadata

* iceberg: keep a staged file whose published record was renamed away

A create answers a conflict when its record was already visible and
another request changed it before the create could back it out. The
cleanup then looked only at the original name, so a record renamed in
between read as absent and the file it still points at was deleted.

S3 Tables now reports that case with a conflict of its own, and both
create paths keep the staged file on it instead of probing the name.
A conflict over a still-hidden record, lost to another create, still
removes the file.

* iceberg: discard a losing create's metadata only while it is still its own

The cleanup deleted the staged file by name. The path can be reused
before it runs: the table is dropped and purged, and another create
stages the same v1.metadata.json and registers it, which the losing
request then deleted.

Staged files now carry a per-request token in an internal extended
attribute, and the cleanup deletes with that token as the filer's
ExpectedExtended precondition, checked under the path lock the delete
runs under. A file that took the path since is left in place.

* iceberg: classify a typed S3 Tables error by its code, not its message

isS3TablesAlreadyExists matched "already exists" in the message before
looking at the code, so an InternalError whose text mentioned an
existing entry read as the name being taken: CreateTable answered 409
instead of 500, and registrationRefused discarded a staged file the
catalog may still point at. A typed error now decides by its code; only
an untyped one is matched by message.

* iceberg: pin stage-create on in the stage-create conflict test

Stage-create is on by default, but a shell that sets
ICEBERG_ENABLE_STAGE_CREATE=false turned the expected 409 into a 501.
Setting it in the test keeps the result independent of the environment.

* iceberg: test two creates overlapping at the canonical metadata name

The concurrent-creators test only lines the creators up before they
start, so whether two of them ever collide at v1.metadata.json is up to
the scheduler. The new test fixes the order: the second create runs
while the first has staged v1.metadata.json and not yet registered it.
It checks that every staging attempt is exclusive, that the second falls
back to v1-<uuid>, that both files hold their own metadata while both
are staged, and that the loser removes only its own file.

* s3tables: a create whose unmark write never went out stayed hidden

confirmChild counted any failure to take a create's mark off as possibly
published, the lookup that precedes the write included. That lookup
failing leaves the mark on, so the record stayed hidden; if another
create then adopted it, the back-out reported a published record that
changed, and the Iceberg catalog kept a staged file no record names.

The mark-removal calls now go through a wrapper that notes whether an
UpdateEntry went out without the filer refusing it. Only then may the
record have been seen. A refused compare, a missing entry, or a failure
before any write leaves the create's record hidden.

* s3tables: pass mayBeVisible into backOutChild

Fixes the build after rebasing onto the backOutChild split.
2026-11-07 10:54:04 -08:00
985706cc32 volume: check a needle's CRC the first time a range of it is read (#12467)
* volume: check a needle's CRC the first time a range of it is read

A ranged read never checked the CRC, so a replica with a damaged needle
served the damaged bytes with 206 and the reader never tried another
copy. The first ranged read of a needle now streams the whole needle
through the existing CRC check, with the usual pooled page buffers,
before writing any byte. A mismatch fails the read, so the reader falls
back to another replica. Needles that pass go into a bounded LRU, keyed
by volume, compaction revision, id, append time and checksum, so later
ranges of the same needle stream as before.

Fixes #5959 for ranged reads.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* volume: key verified ranges by volume load, share concurrent checks

From review: the cache key read v.SuperBlock.CompactionRevision without
the data file lock, racing a compaction commit that reloads the super
block, and a remounted copy of a volume inherited the old copy's
verified entries. Key entries by a load generation instead, an atomic
counter that changes on every load, including the reload in
CommitCompact, and cache a result only if no reload happened during
the check. Concurrent first ranges of a needle now share one check
through singleflight instead of each reading the whole needle.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* volume: keep hot needles verified, leave the io error streak to the range read

From review: Contains does not refresh an entry's place in the LRU, so
even a constantly read needle was evicted after 64k other needles were
verified; look it up with Get instead. The verification pass also
cleared the io error streak before the range read ran, so a range read
failing with EIO after a passing check never built up a streak; the
pass into io.Discard now leaves the streak to the range read, like the
meta read before a streamed GET.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* volume (Rust): check a needle's CRC the first time a range of it is read

Mirror of the Go change: a range served from the data file now checks the
whole needle's CRC before any byte of it is sent, so a damaged replica
answers 500 and the reader tries another copy. Needles that pass go into a
bounded LRU keyed by the volume's load generation, needle id, append time
and checksum; concurrent first ranges of a needle share one check, and a
check that passes leaves the io error streak to the range read.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* volume (Rust): a range over the whole needle checks its CRC

Go reads a range that covers the whole needle through the checked full-read
path. The Rust volume server read it from the data file unchecked, so
"bytes=0-" served a damaged needle with 206. Check the slice's CRC before
answering, as the full read does.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* volume: check the covered blocks of every range against CRCs recorded at the first check

The first ranged read of a needle now records the CRC of each block
(64 KiB to 1 MiB, at most 64 blocks below 64 MiB) while it checks the
whole needle. Later ranges read the block-aligned span around them and
write a block's bytes only once it matches, so damage that appears
after the first check is not served.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* volume (Rust): check the covered blocks of every range against CRCs recorded at the first check

verify_checksum now records a CRC per block (same sizes as Go's
rangeCheckBlockSize) and returns them from the cache; read_checked_range
reads the block-aligned span of a range and answers 500 if a block no
longer matches, so damage after the first check is not served.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
2026-11-07 10:53:07 -08:00
Chris LuandClaude Opus 5.5 d0b5529a2d S3 Vectors: weed-vector, weed-worker and the CLI write tables to the filers and volume servers (#12470)
* sw-client: the filer and volume client library

The client the Enterprise repository's VFS and vector engine use, with
the same API: filer gRPC for entries, listings, volume locations,
assignments and remote caching; volume HTTP, plain or TLS, for needle
bytes; chunk resolution for sparse, overlapping and manifest chunks; a
location cache; and the write side. Enterprise-only protocol fields
(license, conditional mutations, hard links, KV listing) are left out.
Compiled against weed/pb/filer.proto like the other worker crates.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* weed-lance-worker: reads straight from the filers and volume servers

direct_read::DirectReads wraps the object store Lance opens so that
entries, listings and bytes come from the filers (gRPC) and the volume
servers (HTTP, signed with the read JWT), whatever the entry holds:
inline content, chunk manifests, sparse or overlapping chunks, the
filer's volume encryption, the gateway's SSE-S3 (sse.rs, with the KEK
from security.toml or the filer) and objects tiered to remote storage.
Writes still go to the inner store. The code and its tests are those of
the Enterprise repository's engine.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* weed-vector: read tables from the filers and volume servers

The engine asks the masters for the filers (and follows the list), takes
jwt.signing.read.key, grpc.client, https.client and the SSE-S3 keys from
security.toml, and opens every table with DirectReads innermost and the
read cache outside it. --masters is required: the gateway finds the
engine through them and the engine finds the filers. Writes still go to
the gateway.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* sse: chunks out of a manifest keep their own SSE-S3 keys

A chunk expanded from a manifest carries its id as fid, so keying the
per-chunk key cache by file_id gave every such chunk the first one's key
and IV. Keyed by the chunk's id however it is carried.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* direct reads: a silent filer is passed over like an unreachable one

Each filer attempt is bounded, so a filer that accepts the connection and
never answers does not hold the read or the startup lookup.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* sw-client: carry the assignment's fsync to the volume upload

The filer's storage rule may want a needle on disk before the volume
answers; the assignment says so and the upload now asks for it, as the
Go uploader does.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* weed-vector: argument tests pass the required masters; connect timeout for filers

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* direct reads: a failed read is tried once more against fresh locations

A fetch that fails forgets the volume locations it was made against; the
request now looks them up again and retries instead of failing on a
volume that moved.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* sw-client: an exclusive create that finds its entry, and needle deletion

create_entry marks ENTRY_ALREADY_EXISTS with EntryExists, so a caller can
tell a lost conditional create from a failure; the transport deletes a
needle whose entry was never created.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* sse: encode chunks as the gateway stores them

EntryEncoder writes SSE-S3 under a fresh object key wrapped by the KEK,
serialized with the IV, key id and commitment the gateway's reader
expects, each chunk under the IV at its offset; and the filer's volume
encryption under a per-chunk key.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* direct storage: writes, deletes and table removal without the gateway

A put uploads its chunks to the volume servers the filer assigns, at
the filer's chunk size, encoded as the bucket and cluster ask, and
creates the entry the gateway would: md5, mime, size, the ETag
attribute, the SSE-S3 attributes. A conditional create is the filer's
exclusive create, so Lance's manifest commit conflicts the same way;
the needles of a create that failed are discarded. A multipart upload
turns parts into chunks and derives its ETag from them. Deletes remove
the entry with its data, and remove_table a table directory. Copy and
rename, which Lance does not use on S3, are not offered.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* vector.proto: Table says whether its bucket encrypts with SSE-S3

The engine writes the table's files itself, so it needs what the S3 write
path knew: whether the bucket declares SSE-S3. storage_options go: the
engine reaches no S3 store.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* s3vectors: send the engine the bucket's SSE-S3 setting

The rule the S3 write path applies to a Lance bucket's files, factored out
of DeclaredBucketEncryption, decides it from the vector bucket's record.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* weed-vector: every table operation goes to the filers and volume servers

Opens, index creation and drops all go through the direct store, so the
engine has no S3 client of its own left to protect: --storage-option,
the endpoint pin and the request-option allowlist go, and lance's own S3
store is built unsigned and never dialed. The bucket's SSE-S3 setting
comes with the table; -encryptVolumeData encrypts chunks for the volume
servers as the gateway's flag does.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* s3vectors tests: the engine takes no credentials

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* s3vectors e2e: the engine writes what the gateway would have

Every leg checks that the entries the engine wrote carry the gateway's
ETag attribute, the hardened leg that their chunks are encrypted for the
volume servers, and a new test that an SSE-S3 bucket's files are written
under keys the gateway unwraps when it serves them over S3.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* cluster plumbing moves to the worker crate for the engine, worker and CLI

security.toml, the gRPC TLS connector and acceptor, the master dialer,
the filer discovery and the storage connection built from them now live
in weed-lance-worker (cluster, security, grpc_tls), where the worker and
the CLI can use them too. The server crate re-exports security and tls
under their old paths.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* weed-worker, weed-vector-cli: tables through the filers and volume servers

dataset::Storage replaces the S3 credentials every job and the CLI
carried: a table is opened and written through the cluster's storage,
encrypted as its bucket declares (read from the bucket's record on the
filer), and dropped there. Both take --masters and security.toml like the
engine; --access-key and --secret-key go, and with them the catalog's
x-api-key credential, which the cluster token replaces.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* lance catalog: the cluster token alone authenticates a cluster process

The engine, the maintenance worker and the CLI hold the cluster's filer
signing key and reach a table's files themselves. A request presenting
the cluster token and no other credential is theirs: it is authenticated
by the token, runs through the trusted manager, and is vended no S3
credentials.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* worker chart, README and test design follow the worker's new arguments

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* s3vectors tests: the worker and the CLI reach the cluster through the masters

The cluster gets a filer signing key, which the catalog authenticates
the worker and the CLI by; both take --masters in place of S3 keys.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* direct storage: encrypted chunks each start their own CTR stream

The gateway encrypts a multipart part from the IV derived for its offset,
from position zero; the encoder encrypted from the exact byte offset of
the object's stream instead, which agreed only at block boundaries. A
part of 3,000 bytes read back wrong; now tested.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* direct storage: keep what a lost reply saved, discard what a failed put left

A create the filer saved but did not answer, or an exclusive create
retried onto this very upload, keeps its chunks and counts as written;
the needles of a put that failed midway are discarded. The report prints
reads as reads.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* weed-worker: the security.toml it is given also holds its signing key

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* worker chart: the Lance worker mounts grpc.client's certificate

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* direct reads: the filer deadline bounds a listing page, not the listing

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* weed-vector: security.toml booleans read as viper reads them

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* weed-vector: a filer the master lists badly does not hide the others

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* weed-vector: register with the masters only once ready to serve

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* sse: every chunk after the first is written under a key of its own

Chunks of one object shared a key with IVs derived from their offsets, so
two chunks meeting inside a cipher block shared keystream. The first
chunk stays under the object's key and IV, which is how the gateway reads
an object of one chunk; every other chunk gets a fresh key and IV, as the
gateway's multipart parts have and as it reads an object of several.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* worker chart: the Lance worker is given the filer group

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* s3vectors e2e: the gateway serves engine-written SSE-S3 files of one chunk and of several

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* sw-client: the HTTP transport asks the volume to fsync when the assignment does

write_synced fell back to the trait default, which refuses fsync, so every
upload under a fsync storage rule failed before it was sent.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* sw-client: https.client.ca alone vouches for volume servers

With a CA configured, reqwest still trusted the system roots next to it;
weed processes trust only the configured pool.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* sw-client: an empty volume range is read without a request

A zero length underflowed the Range end, and a range past u64::MAX
overflowed it; the first now returns no bytes, the second an error.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* sw-client: a whole-needle answer to a ranged read is cut to the range

A server or proxy that ignores Range answers 200 with the whole needle;
the read returned its first bytes in place of the ones asked for.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* sw-client: a read size past u64::MAX reads to the end of the file

offset + size overflowed before the clamp to the file size.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* sw-client: an SSE-encrypted manifest is refused without a decoder

Only filer-encrypted manifests were; an SSE one went to the protobuf
decoder as ciphertext.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* sw-client: an empty write leaves the staged file size alone

A zero-byte write past the end grew the size to its offset, as a
zero-byte pwrite does not.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* sw-client: the chunk cache keeps no empty chunks

They add no bytes, so nothing evicted them and the map grew with every
distinct empty needle, even at capacity zero.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* sw-client: flushing with a zero chunk size is an error, not a panic

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* weed-vector: an empty master list fails at startup

"" split into one empty address that passed the check, and the engine
then waited two minutes for a filer before a generic error.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* lance cluster: an empty master list fails at startup

Cluster::new took "" as one master, so weed-vector, weed-worker and the
CLI waited two minutes for a filer before a generic error.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* direct storage: a failed create keeps its chunks when the filer cannot be asked

The lookup after a failed create is what tells a lost reply from a
refusal; when it fails too, the entry may be saved, so its chunks stay.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* weed-vector: a gateway-described table takes the shared unsigned store options

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* vector.proto: give sse_s3 a new field number and reserve storage_options

Field 4 held the storage_options map on master; a gateway from that build
would send a length-delimited value the engine cannot decode as a bool.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* weed-vector-cli: open s3:// URIs on the cluster's storage

A table named by its s3:// location skipped the store wrapper and opened
Lance's own S3 store, which has no credentials since the fallback flags
were dropped. It now opens like a namespace table and needs --masters;
local paths keep the plain builder.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* vector: an index id is plain path components, and drop trusts describe()

`resolved()` now refuses an id whose part is empty, `.`, `..` or holds a
`/`, since the colocated path would land elsewhere on the filer; and
`drop_index` no longer repeats the colocation check for the location
describe() already enforced for the configured schemes, which made a drop
on a memory:// table fail in the scheme test configuration.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* vector: a file delete takes only its entry; a table path is plain names

The store's delete of one file no longer asks the filer for a recursive
delete, so a path that names a directory does not take its content with
it, and `remove_table` refuses a location whose bucket or path component
is empty, `.`, `..` or holds a separator, which the filer would resolve
outside the table.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* vector: chunks are assigned to the bucket's collection

The filer assigns a chunk's volume from the collection the caller names;
the gateway names the bucket, prefixed by the filer group when the
cluster has one, so a plain assign landed the engine's chunks in the
default collection. The store now asks for `<bucket>` or
`<group>_<bucket>`, with the group taken from the cluster connection.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* vector: a put's content type becomes the entry's mime

Lance sets no content type today, so every entry was written as
application/octet-stream; a put that does carry one now creates the
entry with it, as the gateway does.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* vector: an upload whose create went unanswered is done with

A create the filer took and never answered (a timeout, or a transport
that reports the filer unavailable) may still be applied, so its chunks
are kept, as they already were for a create whose reply was lost. The
upload then records that it is done: a second completion is refused
instead of creating an empty entry, and an abort discards nothing.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* vector: an upload dropped unfinished discards its chunks

A request cancelled mid-write drops its multipart upload with neither
complete nor abort called, leaving its needles on the volume servers.
Dropping an upload that still holds chunks now discards them on the
runtime; a completed upload, or one whose completion is in doubt, holds
none.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* s3: a cluster process registers and deregisters in the vector namespace

The Lance catalog ran a cluster-token caller through the trusted manager
for every operation but the vector-namespace write check of register and
deregister, which still used the caller-checked manager and answered 403
for an index of a LANCE bucket. One `manager(r)` now picks the manager
for all three.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* vector: the worker's flags take two dashes

The worker parses its flags with clap and, unlike the engine, does not
accept Go-style single-dash long flags, so the harness's hardened worker
leg and the README's `-encryptVolumeData` did not start it. The README's
commands also gain the `--masters` the worker now requires.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* vector: the CLI takes --encryptVolumeData

An index build or optimize from the CLI wrote its chunks unencrypted for
the volume servers unless the filer itself asked for the cipher; the CLI
now takes the flag the engine and the worker have.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* vector: a bucket's encryption is read when a table is opened

The setting was cached for a minute, so a bucket recreated with SSE-S3
within that minute had its first tables written plain, and a record that
could not be decoded was read as "no encryption". A table open now reads
the bucket's record from the filer, once per open, and an unreadable
record is an error.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* vector: a removed table's files leave the object cache

`remove_table` asked the cache to forget everything under the table, but
the cache was built without invalidation closures, so the call was refused
and its error dropped: a data file read before the drop was still served
afterwards. The cache now supports the invalidation, and a refusal is an
error.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* vector: an abort keeps the chunks it could not discard

`discard_all` stopped at the first chunk a volume server would not give
up and dropped the rest from the upload's state, so a retried abort found
nothing to do and the needles stayed. Every chunk is now attempted, the
ones left are kept for the next abort or the drop, and the upload can no
longer be completed.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* vector: a chunk write that fails gives its needle back

When the volume server took the chunk but its answer was lost, the needle
stayed under a file id nothing would ever reference. The assignment is
known before the write, so a failed write now discards it, best effort,
before the error is returned.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* vector: the live worker tests require the cluster's masters

With WEED_LANCE_NAMESPACE set and WEED_MASTERS not, the tests seeded their
tables through lance's own S3 store, which reaches nothing, and failed
before a job ran. The missing variable is now named outright.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* direct storage: only the filer's missing-entry error means not found

Matching any "not found" text read failures such as an unresolved remote
storage as a missing entry, so a delete or a listing reported success over
data that was still there.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* direct storage test: compare the chunks' SSE-S3 IVs

Each chunk's metadata wraps its key under a fresh nonce, so the metadata
differed even if the key and IV were reused.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* direct storage: a filer passed over unanswered keeps the outcome open

When one filer timed out on a create and the next refused it, the refusal
came back without the no-answer mark, so the put discarded chunks the
first filer may still reference.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* lance test: start every concurrent writer before awaiting them

The lazy map spawned each writer only after the previous one finished, so
the commits never overlapped.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* direct storage: count failed uploads and deletes

The storage report counted failed reads and lookups but not a volume
upload, an object delete or a table removal that failed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* S3 Vectors tests: run weed-vector-cli as the cluster's processes run

The CLI ran without the fixture's environment and outside its directory,
so under VECTOR_READ_JWT it could not read the volumes and under
VECTOR_TLS it found no security.toml.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* direct storage: a filer that fell away leaves the outcome open too

An unavailable error (connection reset, deadline, transport failure) can
arrive after the filer took the request, so a refusal from the next filer
no longer settles a create the first one may still apply: the chunks are
kept, as after a timeout.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* direct storage: a chunk write abandoned in flight gives its needle back

The needle is assigned before the write, so a request cancelled while the
write is in flight left it on the volume server with nothing to name it.
`Source::upload` is now `assign` and `write`, and the upload holds the
assignment across the write in a guard: a write that fails discards it
and one abandoned discards it on the runtime, while a write that returns
hands it to the entry.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* lance tests: sign for the namespace as the worker does

The live tests reached the namespace anonymously once the S3 keys went,
and an anonymous caller lists no tables, so detection proposed nothing.
They now take the cluster key from where the worker reads it
(WEED_JWT_FILER_SIGNING_KEY or security.toml) and log at info, so a table
a handler skips says why in the harness output.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* s3tables lifecycle test: run the Lance worker against the cluster

The worker leg of TestLanceTableLifecycle still handed the Rust tests S3
credentials and no masters. The mini cluster now has a cluster key, and
the tests get WEED_MASTERS and that key; passes with
WEED_LANCE_MAINTENANCE=worker (3 fragments became 1, 4 versions removed).

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* direct storage: name no collection outside a filer group, as the gateway

Without a filer group the gateway leaves the collection to the filer, so a
filer.conf rule for the bucket's path picks it; naming the bucket
overrode that rule, and the chunks landed where a bucket delete does not
look.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* direct storage: a read reports the entry's content type

A put now stores its ContentType as the entry's mime, but get_opts
returned no attributes, so the type did not come back on a read.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
2026-11-07 10:51:52 -08:00
Chris LuandClaude Opus 5.5 5cd38527ba filer.remote.sync: skip pre-mount mount-table versions, decide the unmount before admitting later events (#12472)
* filer.remote.sync: judge the mount table before admitting later events

The unmount was decided inside a processor job, so events behind the
mapping write, such as remote.unmount's purge deletes, were admitted and
could delete remote objects before the exit landed. The stream callback
now reads each mount-table version before the next event, admits nothing
after the one that ends the mount, saves no offset, and drains the
admitted jobs before exiting.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* filer.remote.sync: ignore mount-table versions from before the mount

Without a saved offset the follow starts at the mounted directory's
creation second, so it replays mount-table versions written before
remote.mount added the directory. Each read as an unmount and the
process exited on every start. A version now ends the write-back only
when the version it replaced mapped the directory to its location.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* filer.remote.sync: judge the mount table by the mount's own key

-dir may lie below a mount, and DetectMountInfo resolves it to the
enclosing mount. Looking the table up by -dir found no mapping in any
version, so a replaced table never counted as mapped and the parent's
unmount was missed, admitting its purge deletes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* filer.remote.sync: a table renamed away ends the mount

A rename of /etc/remote/mount.mapping to another name inside /etc/remote
was ignored, although the filer no longer has a mount table. The table is
now matched by its exact path on both sides of the event, so renaming it
away counts as removing it, and a file of that name elsewhere under
/etc/remote no longer counts as the table.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* filer.remote.sync: a mount on another remote is not the same mount

The mount-table check compared only bucket and path, so remounting the
directory on another remote storage at the same bucket and path kept
writing through the old client, and a replayed unmount from the other
remote stopped the write-back. The remote storage name now counts too.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* filer.remote.sync: check the drain at the exit itself

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* filer.remote.sync: resolve the mount covering -dir in each table version

Judging the table by the startup mount key alone missed a mount added at
-dir itself below that key: the key stayed mapped, so the nested mount and
its later removal were both admitted, and the removal's purge deletes ran
against the startup remote. Each version is now resolved the way
DetectMountInfo resolves the live table, and a different covering mount
ends the follow as a move.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* filer.remote.sync: an earlier mount's unmount still ends the follow

Skipping every version whose replaced table did not map the directory to
this mount also skipped the unmount of an earlier mount of the directory,
which a -timeAgo replay reaches, and admitted its purge deletes against
this mount's remote. Only a version in which no mount covers the
directory, replacing one in which none did, is now skipped as history
from before the mount.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* filer.remote.sync: map paths below -dir under its place in the mount

With -dir below a mount, every remote key was taken relative to -dir and
appended to the mount's location, so /mount/sub/x.txt uploaded to x.txt
instead of sub/x.txt and could overwrite another object. The processor
now gets the remote location of -dir itself; the mount's own location
still judges mount-table changes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* filer.remote.sync: resume below a mount no earlier than the mount's offset

remote.unmount saves the offset of the directory it unmounts, after its
purge, and leaves the offsets of directories below it alone. A
write-back of -dir=/mount/sub resumed from its own offset after a
remount, replayed the unmount and exited, on every start. Without
-timeAgo it now starts no earlier than the mount's offset. Whatever that
skips was purged, or was synced by a write-back of the whole mount.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-07 05:06:52 -08:00
Chris LuandClaude Opus 5.5 5b1450b06b s3tables, iceberg: one catalog record per table location (#12473)
* s3tables: refuse a table or view location another record holds

A create, register, update or rename could name a location at, under or
above another catalog record's files, including the files a renamed table
left under its old name. The new record then loaded that table's metadata,
had credentials vended for its files, and could purge them on drop.

Each of those writes now scans the bucket's records and refuses a root
another record claims, and a drop of a record sharing its files is refused.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* iceberg: check a table's location before writing to it

REST create and stage-create write a metadata file, and stage-create
vends write credentials, before the catalog registers the table, so the
create gate now also validates the location and refuses one another
record holds. RegisterTable runs the same gate before it reads the
metadata file, instead of reading first.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* iceberg: keep a dropped location another record holds

DropTable and DropView purge the dropped record's location after the
catalog delete. A view recorded at a table's location took that table's
directory, record included. The purge now leaves a location another
record still holds.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3tables: authorize CreateTable before reporting the namespace

A caller without CreateTable got 404 for a missing namespace, 403 for an
existing one and 500 with decoder text for an unreadable one. The grant is
now evaluated against the bucket first, and only a caller that passes
learns the namespace is missing or unreadable.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* iceberg maintenance: skip a table whose files another record shares

Orphan removal and snapshot expiry delete what a table's metadata does not
reference. For two records over one location that deletes the other's
files, so detection leaves such tables alone.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3tables: count another record's catalog entry as claimed

A table keeping its files elsewhere still owns its entry directory, and a
view recorded there would take that entry with it on drop. The ownership
check now refuses a root at, above or under another record's entry too.
Maintenance still compares files only, so a table created over a renamed
table's old name does not stop that table being maintained.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3tables: check a claimed location again once the record is written

Two creates, or a create and an update moving a root, could both read the
same root free and both land. A create now checks again after writing its
still-marked record and backs out if another record claims the root, and an
update that moved its root checks again after the write and puts the previous
metadata back. Of two racing writers at least one sees the other.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* iceberg: enable stage-create in the location ownership test

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* iceberg maintenance: check for a shared location again before removing orphans

A job can run long after its table was scheduled, or come from a detector
that never checked, so remove_orphans reads the bucket's records itself and
leaves the files alone when another record shares the table's location.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3tables: check a renamed record's new root again once it is written

A record without a location moves its root with its name, and was checked
only before the rename wrote the destination, so a create landing on that
root in between could share it. The rename now checks again with the
destination written and backs out, like a create.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3tables: put a moved record back when its location cannot be checked again

An update that moved its root kept the move when the check after the write
failed to list the bucket, so a racing writer that relied on that check could
share the root. The move is now undone as it is when the root is taken.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3tables: hold a moved record's old root until the move lands

An update moving a record's root wrote the new metadata first and put the
old metadata back when the check that followed found the new root taken.
The old root was free in between, so a create racing for it could pass its
own check and then share it with the restored record.

The update now stores the metadata it is about to write beside the current
one, so the record holds both roots, checks the new root with that
reservation in place, and only then switches the metadata. A move that
backs out drops the reservation without touching the metadata, and an
update that keeps its root clears a reservation a crashed move left behind.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3tables: test a move racing another update of the record

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3tables: drop a move's reservation when a rename or delete takes the record

A rename or a decoupled delete that took a record while an update held its
move reservation left the reservation on the old name, where the move could
no longer clear it, and later creates of that name were refused as taken.
Both now clear it with the record's other attributes, and a failed rename
does not put it back.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test: register a table from metadata under its own location

The Spark register test registered a live table's metadata file under a
second name, which now fails because both records would own the source's
files. It checks that refusal instead, and registers a copy uploaded under
the new table's own directory, with its location pointing there, to cover
reading and writing a registered table.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-07 05:04:09 -08:00
Chris LuandClaude Opus 5.5 b650303aee S3 Vectors: weed-vector reads tables from the filers and volume servers (#12469)
* sw-client: the filer and volume client library

The client the Enterprise repository's VFS and vector engine use, with
the same API: filer gRPC for entries, listings, volume locations,
assignments and remote caching; volume HTTP, plain or TLS, for needle
bytes; chunk resolution for sparse, overlapping and manifest chunks; a
location cache; and the write side. Enterprise-only protocol fields
(license, conditional mutations, hard links, KV listing) are left out.
Compiled against weed/pb/filer.proto like the other worker crates.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* weed-lance-worker: reads straight from the filers and volume servers

direct_read::DirectReads wraps the object store Lance opens so that
entries, listings and bytes come from the filers (gRPC) and the volume
servers (HTTP, signed with the read JWT), whatever the entry holds:
inline content, chunk manifests, sparse or overlapping chunks, the
filer's volume encryption, the gateway's SSE-S3 (sse.rs, with the KEK
from security.toml or the filer) and objects tiered to remote storage.
Writes still go to the inner store. The code and its tests are those of
the Enterprise repository's engine.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* weed-vector: read tables from the filers and volume servers

The engine asks the masters for the filers (and follows the list), takes
jwt.signing.read.key, grpc.client, https.client and the SSE-S3 keys from
security.toml, and opens every table with DirectReads innermost and the
read cache outside it. --masters is required: the gateway finds the
engine through them and the engine finds the filers. Writes still go to
the gateway.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* sse: chunks out of a manifest keep their own SSE-S3 keys

A chunk expanded from a manifest carries its id as fid, so keying the
per-chunk key cache by file_id gave every such chunk the first one's key
and IV. Keyed by the chunk's id however it is carried.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* direct reads: a silent filer is passed over like an unreachable one

Each filer attempt is bounded, so a filer that accepts the connection and
never answers does not hold the read or the startup lookup.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* sw-client: carry the assignment's fsync to the volume upload

The filer's storage rule may want a needle on disk before the volume
answers; the assignment says so and the upload now asks for it, as the
Go uploader does.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* weed-vector: argument tests pass the required masters; connect timeout for filers

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* direct reads: a failed read is tried once more against fresh locations

A fetch that fails forgets the volume locations it was made against; the
request now looks them up again and retries instead of failing on a
volume that moved.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* direct reads: the filer deadline bounds a listing page, not the listing

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* weed-vector: security.toml booleans read as viper reads them

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* weed-vector: a filer the master lists badly does not hide the others

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* weed-vector: register with the masters only once ready to serve

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* sw-client: the HTTP transport asks the volume to fsync when the assignment does

write_synced fell back to the trait default, which refuses fsync, so every
upload under a fsync storage rule failed before it was sent.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* sw-client: https.client.ca alone vouches for volume servers

With a CA configured, reqwest still trusted the system roots next to it;
weed processes trust only the configured pool.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* sw-client: an empty volume range is read without a request

A zero length underflowed the Range end, and a range past u64::MAX
overflowed it; the first now returns no bytes, the second an error.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* sw-client: a whole-needle answer to a ranged read is cut to the range

A server or proxy that ignores Range answers 200 with the whole needle;
the read returned its first bytes in place of the ones asked for.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* sw-client: a read size past u64::MAX reads to the end of the file

offset + size overflowed before the clamp to the file size.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* sw-client: an SSE-encrypted manifest is refused without a decoder

Only filer-encrypted manifests were; an SSE one went to the protobuf
decoder as ciphertext.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* sw-client: an empty write leaves the staged file size alone

A zero-byte write past the end grew the size to its offset, as a
zero-byte pwrite does not.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* sw-client: the chunk cache keeps no empty chunks

They add no bytes, so nothing evicted them and the map grew with every
distinct empty needle, even at capacity zero.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* sw-client: flushing with a zero chunk size is an error, not a panic

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* weed-vector: an empty master list fails at startup

"" split into one empty address that passed the check, and the engine
then waited two minutes for a filer before a generic error.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
2026-11-07 04:21:13 -08:00
Chris LuandClaude Fable 5.1 2fe480a6e2 S3 Vectors: the gateway resolves the table it sends to weed-vector (#12468)
* lance catalog: in-process DescribeTable for trusted callers

DescribeTable and DescribeTableInternal resolve a table's location and
storage options without the HTTP layer; the internal form runs the lookup
through a trusted manager so the caller's catalog grants do not gate a
resolution its own permission already authorized. Same code as the
Enterprise repository.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* vector.proto: Table carries its location and storage options

The gateway resolves the table it addresses, so the engine can open it
without a describe round trip of its own. Both fields are optional; an
empty location means the engine looks the table up as before.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* lance catalog: test the in-process describe

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* s3vectors: resolve the table before calling the engine

The gateway describes the table in process, through the trusted catalog
manager, and sends its location and storage options with every engine
call. Storage options are vended for the caller's principal, as the
Enterprise router does for queries. A catalog failure answers with the
code the engine would have given.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* weed-vector: open a table at the location the gateway sent

A request whose table carries a location is opened there, with the
storage options the gateway vended over the engine's fallback; the
catalog is described only for a table that comes without one, as from
an older gateway. dataset::open_at and dataset::drop_at take a
description the caller already holds.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* s3vectors e2e: run the engine without a catalog

VECTOR_ENGINE_NO_CATALOG=1 gives the engine an unreachable catalog URL;
every CI leg sets it, so the suite proves the gateway resolves each table
it sends.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* lance catalog: an internal describe may write a vector index

With credential vending on, the options vended to the engine followed the
caller's s3tables grants, which never allow writing the vector namespace,
so every write the gateway sent would have run with read-only credentials
and reads would have lost their version lease. The engine is that
namespace's writer and the caller's vector permission authorized the
operation, so an internal describe of a vector table vends write-capable
options; other namespaces keep the UpdateTable check.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* s3: resolve tables for the engine only while the catalog runs

With -port.lance=0 the gateway has no catalog to resolve through; the
engine then keeps looking tables up itself, as before.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* lance catalog: a deregistered table describes as NoSuchTable

So the gateway answers NotFoundException for it, not InternalFailure.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* weed-vector: trust a sent location only for a colocated vector index

A caller that reaches the engine could otherwise name any storage the
engine's credentials cover. A location is taken from the request only
when the id names a vector index and the location is its colocated path,
which the engine writes already; anything else is described through the
catalog as before.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* weed-vector: scope the read cache by storage endpoint

The cache was keyed by bucket and path alone, so the same path reached
through another endpoint would have shared entries. Each open wraps the
store with the cache scoped to the endpoint its options name.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* weed-vector: keep a configured endpoint over one a request names

A caller that reaches the engine could otherwise send its fallback
credentials to an endpoint of the caller's choosing. When the engine's
--storage-option names an aws_endpoint, a table whose request names
another one is described through the catalog instead.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* weed-vector: use request options only where credentials cannot leave

A request's storage options are taken only when they hold the keys a
gateway's describe produces, so a proxy or an endpoint alias cannot route
the engine's credentials elsewhere, and an endpoint only when it is the
configured one or the engine holds no credentials to send. Anything else
sends the table to the catalog.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* s3vectors e2e: give the engine its endpoint with its credentials

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* weed-vector: a drop checks the request's options like an open

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* weed-vector: an unpinned endpoint needs the request to sign for itself

Without configured credentials the S3 store still signs with the AWS
default chain, so an endpoint from a request is taken only when the
request skips signing or brings its own credentials.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* weed-vector: a trailing slash does not unpin the endpoint, and say so

A configured aws_endpoint written with a trailing slash matched nothing
the gateway sends, so every table fell back to the catalog in silence.
The comparison ignores the slash, and a fallback for a resolved table is
logged at debug.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* weed-vector: resolved by the gateway, or refused

The engine no longer has a catalog: --namespace and --filer-signing-key
are gone, a request whose table names no location is InvalidArgument, and
one whose location is not the index's own or whose storage options the
engine's credentials must not be used with is PermissionDenied. A drop
removes the colocated path it was given.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* s3: the catalog resolves vector tables whether or not its port is served

newLanceServer builds the Lance catalog for the S3 Vectors API; the
listener on -port.lance is optional.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* s3vectors tests: start the engine without a catalog

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* s3: the aux listener test builds the catalog before serving it

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* weed-vector: an unpinned endpoint takes only credentials the request signs with alone

Lance fills an option a request leaves out from the engine's environment,
under any of object_store's names for it (TOKEN, SESSION_TOKEN, ...), and
asks the AWS default chain when the key pair is incomplete. A request that
named its own endpoint with only an access key id, or a key pair while the
engine's environment held a session token, would have sent the engine's
credentials there. Such a request now needs a full key pair, and its own
session token whenever the environment would supply one.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* weed-vector: accept --namespace and --filer-signing-key, ignored

Both flags shipped in 4.52 and are documented, so dropping them made an
existing command line exit at startup with an unexpected-argument error.
They are kept hidden and ignored, with a warning.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3vectors design: the engine is started without a catalog

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* weed-vector: the trust check takes the environment it reads

The options test read the process environment through signs_alone, so a
host exporting AWS_SESSION_TOKEN or TOKEN failed it. The caller now
passes the variable names in; describe passes the engine's own.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* weed-vector: log both endpoints when a request's options are refused

The gateway turns the PermissionDenied into a bare AccessDenied and logs
nothing, so a pinned aws_endpoint written in another form than the one the
gateway sends (host case, a default port, an IP for a name) failed every
call with no trace of which value differed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3: the Lance catalog sets the credential role whether or not Iceberg runs

-iceberg.credentialRole reached the S3 server only while the Iceberg
listener started, so with -port.iceberg=0 the Lance catalog installed its
vendor but vending stayed off, and the vector engine got no credentials for
the tables the gateway resolves. Both catalogs now set the role through
one helper.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3: set the credential role once, before either catalog serves

Both catalogs wrote the role as they were built; the second write landed
while the first was already serving requests that read it.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

---------

Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
2026-11-06 19:39:09 -08:00
Chris LuandClaude Opus 5.5 7beac0e8ea admin: show disk usage on the dashboard instead of the data size (#12466)
* admin: show disk usage on the dashboard instead of the data size

The dashboard's Total Size card summed the volume and EC shard sizes the
volume servers report. That leaves out deleted needles waiting for vacuum,
EC generations kept after a vacuum, and anything else on the disks, so on a
cluster with a lot of EC vacuum activity it showed far less than the disks
held.

The card now shows the bytes in use on the volume servers' disks, the same
statfs numbers the volume server list uses, with the disks' capacity, and
its trend line follows that number. Servers that report no statfs numbers
count their data size, as on the volume server list.

Claude-Session: https://claude.ai/code/session_011UpvafcfUuC2mLxNAXLQJi

* admin: compute the disk usage percentage in float64

current*100 overflows int64 past about 92 PB of used space, so the
dashboard card showed a negative percentage.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* admin: size the volume server disk bar with calculatePercent64

int(host.DiskUsage) truncates byte counts on 32-bit builds, so the bar
width was wrong for any disk over 2 GB there.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* admin: test the cluster disk totals across statfs and fallback servers

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-06 19:25:10 -08:00
Chris LuandClaude Opus 5.5 fd81a50c57 mount: keep retrying writeback chunk uploads through a volume outage (#12465)
* mount: keep retrying writeback chunk uploads through a volume outage

With -writebackCache, close() returns before the data is uploaded. When
the volume servers stay unreachable past the upload's own retry (~20 s),
the chunk was dropped and the file never appeared, with no error to the
application. Retry transient upload errors with the same bytes until they
succeed, unless the dirty pages were discarded. Permanent errors and the
synchronous mode keep failing as before.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* mount: test writeback upload retries and the cases that still fail

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* operation: keep the filer's read-only verdict on assign failures

UploadWithRetry flattened AssignVolumeResponse errors into a string, so
callers could not tell a read-only rule from an outage. Wrap the filer's
coded error so errors.Is(err, filer_pb.ErrReadOnly) works; the message is
unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* mount: retry writeback uploads the upload layer reassigns, never read-only

A filer proxy that cannot reach the volume answers 500 with an empty
body, which util.IsTransientError does not match, so that outage still
dropped the chunk. Also retry what operation.ShouldReassignUpload takes.
A read-only rule stays final even when its path or message contains a
transient word such as "unavailable".

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* mount: stop a writeback upload once its dirty pages are discarded

Destroy() was only seen after a failed attempt, so a truncate during the
retry wait still started another upload, and a chunk that finished after
its pages were replaced was attached to the new content. Check before
each attempt, and drop the chunk instead of attaching it; volume.fsck
reclaims it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* mount: keep Destroy from slipping between the discard check and the append

truncateEntry destroys the pages and then installs the new entry, so a
Destroy landing after the check but before AddChunks put the old chunk
into the replacing entry. Hold one lock across the check and the append,
and take it in Destroy while setting the flag.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* mount: leave the Windows interrupt drain to the capped filesystem hook

Interrupt hooks run in reverse registration order, so the Windows hook ran
before the filesystem hook and waited on pending writeback flushes without a
bound. With writeback uploads now retrying through an outage, Ctrl-C could
wait until the volume servers came back. Unmount only, as on Linux, and let
the filesystem hook drain for at most 30 s and keep the write cache.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* mount: test a writeback upload whose assign keeps timing out

With every volume server gone the assign RPC times out until the filer
client gives up; that error must be retried like a refused upload.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-06 19:24:42 -08:00
github-actions[bot] 7d6680c7a4 docs: regenerate star history chart 2026-11-07 00:47:27 +00:00
Chris LuandClaude Opus 5.5 99539484b6 grafana: chart the remote-deletion tombstone gauges (#12444)
* grafana: chart the remote-deletion tombstone gauges

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* grafana: describe the fetch hold that outlasts a full tombstone table

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* grafana: word the tombstone panel description like the gauge help

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-06 15:22:38 -08:00
Chris LuandClaude Opus 5.5 1c4fe61fcd master: stop the resumed single-master leader from checking quorum every 1 ms (#12463)
* deps: bump seaweedfs/raft so the leader quorum check follows the election timeout

A single master resuming raft state self-elects on a 1 ms election
timeout and restores the configured one afterwards, but the leader kept
running its quorum check every 1 ms for the rest of the process. The
raft change re-arms that check from the current timeout.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* deps: bump seaweedfs/raft to the raft PR head with the extended test

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* deps: bump seaweedfs/raft for the non-positive election timeout guard

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* deps: bump seaweedfs/raft so a timeout change re-arms the leader quorum check

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* deps: pin seaweedfs/raft v1.2.4

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-06 15:22:06 -08:00
Chris LuandClaude Opus 5.5 cdea7e79e0 filer: fold small chunks into manifests 256 at a time (#12462)
* filer: fold small chunks into manifests 256 at a time

Every metadata event carries the file's whole chunk list, so a file grown
by small appends re-sent each chunk record on every append until the
list reached ManifestBatch (10000). Chunks under 1 MiB now fold once 256
of them build up; full-size chunks still fold at ManifestBatch, so large
files keep their flat lists and read without a manifest fetch.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* filer: merge appended small chunks at the small-chunk fold count

The append path merges small chunks into full-size ones, but only at
1000 of them; the fold now hides small chunks in a manifest at 256, so
the merge would never see enough of them. Merging at the same count
keeps rewriting them into full chunks first.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-06 13:41:02 -08:00
Chris LuandClaude Opus 5.5 728b476ba2 filer.remote.gateway: keep the remote object while a version under a delete marker is locked (#12459)
* filer.remote.gateway: keep the remote object while a version under a delete marker is locked

Every version of a versioned bucket uploads to the object's key, so on a
remote without versioning that object is the only remote copy of the
latest version. A delete marker deleted it even when a version still held
a legal hold or unexpired retention. Keep it while any version of the key
is locked, the same rule the filer uses before pruning a remote entry.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* filer.remote.gateway, filer.remote.sync: say that earlier versions need remote versioning

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* filer.remote.gateway: delete a kept delete marker's object once the version lock is released

A delete marker kept for a locked version stays unstamped. When an
update or delete leaves a version that carried a legal hold or retention
without an active lock, look up the key's latest version and, if it is
still that unstamped marker, run the delete again. Only changes to
versions that carried a lock trigger it, so the noncurrent mark and tag
updates of unlocked versions do not send repeat remote deletes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* filer.remote.gateway: test the kept delete marker retry through the bucket mount

The retry reads the key's versions through a filer client parameter, as
the rename helpers do, so a test can check the mapped object key and the
marker stamp, and that lock-releasing updates and deletes reach it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* filer.remote.gateway: check the lock release lookup goes to the gateway's filer

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-06 13:37:08 -08:00
Chris LuandClaude Opus 5.5 8c22336944 filer.meta.backup: stream from the start of an unfinished walk (#12458)
* filer.meta.backup: stream from the start of an unfinished walk

A walk only inserts entries, so when the first walk is interrupted the
next start walks again and streams from the second walk's start: deletes
and renames between the two walks never reach the backup store. Keep the
first walk's start time in the store until a walk completes and stream
from it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* filer.meta.backup: start fresh once the stream has passed an earlier walk's start

A walk start left in the store by a crash between saving the offset and
clearing it, or by an interrupted -restart followed by a plain restart,
stops mattering once the stream saves an offset past it. Ignore it then
instead of replaying from it on the next walk.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* filer.meta.backup: fail the walk on an unreadable offset

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-06 13:32:46 -08:00
jsasandjames 5ba97b4432 filer: a full remote-deletion tombstone table keeps the lazy fetch off (#12440)
* filer: a full remote-deletion tombstone table keeps the lazy fetch off

At the cap a new delete under a remote mount was dropped with a log line
and isRemoteDeletionPending answered false for it, so the lazy remote fetch
resurrected deleted paths. While the table is full every path may be a
pending delete: isRemoteDeletionPending answers true until the table has
room again. The size and the full state are gauges
(filer_remote_deletion_tombstones, filer_remote_deletion_tombstones_full).

(cherry picked from commit f3be0ab965cde3bfd3053d3ac720fbbe3f68e98b)

* filer: a full tombstone table drains on reads and lifts after the unrecorded deletes age out

While the table is full, a lookup miss evicts expired tombstones and those
the write-back offset has passed, so the table drains without a mutation.
The full state lifts only once the table has room and the newest
unrecorded delete is past the TTL: an unrecorded delete has no event to
confirm it, so only the TTL makes it safe, as for a pending tombstone.

(cherry picked from commit 19394fe5d67798af7b0d755ac18c83c75e20a12b)

* filer: a full tombstone table drains per mount, only when its offset advances

Offset-based release applies to the tombstones under the mount whose
offset was read; another mount's tombstones answer to their own offset. A
read sweeps the table only once that offset advanced since the last drain
for the mount or the unrecorded deletes aged past the TTL, so a listing of
many missing children does not rescan the table per child.

(cherry picked from commit 9cf00e2e54d70e6ba530e6ec1b1ff4b502873847)

* filer: the tombstones-full gauge help names when it clears

---------

Co-authored-by: james <james@medable.com>
2026-11-06 13:28:47 -08:00
7fc33e7fe3 filer: no lazy remote fetch inside a store transaction (#12439)
* filer: no lazy remote fetch inside a store transaction

A rename onto an existing target deletes the target inside the store
transaction, then CreateEntry's FindEntry misses the row and runs the lazy
remote fetch, which persists the archive copy on a second connection. The
store wrapper drops that call's deadline, so the UPDATE waits without bound
on the row lock of the transaction's own uncommitted DELETE while the
rename goroutine holds the path's exclusive locks: every later lookup,
rename and delete of the path on that filer waits forever.

A lookup whose context carries the store transaction skips the fetch.

(cherry picked from commit b69e9535c2597d0c8e856f031d3778fddfd4159c)

* filer: materialize remote-only rename paths before the transaction

With the lazy fetch off inside a store transaction, a source or target the
filer holds only on the remote would read as absent to AtomicRenameEntry
and StreamRenameEntry: a rename with FailIfTargetExists could overwrite it,
and a remote-only source would skip the deferred copy. PrefetchRemoteEntries
runs the fetch for both paths before BeginTransaction, under the path locks.

(cherry picked from commit e61f9718018669ef59768cd5eda25b6466d69597)

* filer: skip the lazy remote fetch only in CreateEntry's lookup inside a transaction

The guard on every lookup inside a store transaction hid a source or target
the filer holds only on the remote from the rename, and the prefetch before
the transaction did not reach the children of a directory rename. The
deadlock needs a write: CreateEntry's lookup after the transaction deleted
the path. That lookup alone skips the fetch; reads inside the transaction
keep it, and their persist on a second connection holds no lock the
transaction waits for.

(cherry picked from commit f555092c324780a8fe5314b238694551e3ef6be5)

* filer: mark a store transaction independently of the store's own key

ArangoDB and FoundationDB keep their transaction under other context
keys, so the CreateEntry guard keyed on "tx" missed them. The filer now
marks every context returned by BeginTransaction and the guard checks
that marker.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
(cherry picked from commit bd0505eb80)

* filer: keep the remote check for an exclusive create in a transaction

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
(cherry picked from commit b6aac15c3b)

* filer: test that a create outside a transaction still consults the remote

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
(cherry picked from commit 972daddd84)

---------

Co-authored-by: james <james@medable.com>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-06 13:21:37 -08:00
Chris LuandClaude Opus 5.5 86db60b669 s3api: mint a versioned PUT's id at commit, after the lock wait and the upload (#12454)
* s3api: mint a versioned PUT's id after the object write lock

putVersionedObject minted the version id before the upload and the lock
wait. A DELETE that committed while the PUT waited got a newer id, so a
conditional PUT whose If-None-Match passed against the delete marker was
stored behind it: accepted, but GET returned NoSuchKey.

The finalize now re-mints the id once the lock is held and the
precondition passed, so the version orders after everything the check
saw. An ambiguous routed attempt keeps its id, since recovery looks for
that entry.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3api: mint a routed versioned PUT's id after the upload

On the routed path the version id was still the one minted before the
upload, and RECOMPUTE_LATEST picks the newest name. A DELETE that
committed during a long upload left its marker latest over the accepted
PUT. Re-mint right before the routed transaction, as the lock path does.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-06 13:20:36 -08:00
1176f975b3 filer: release metadata log chunks a failed append leaves unreferenced (#12455)
* filer: stop unbounded metadata-log growth on append-only files

Two independent leaks make a filer metadata log grow far faster than the
data it describes. On a cluster whose live logs weigh a few hundred
megabytes, they produced tens of gigabytes of metadata per day.

1. Metadata events embed the whole old and new entry, chunk lists
   included. An append-only file — a log or a journal written in small
   pieces — therefore re-serializes every chunk it ever had on every
   append, so the cost is quadratic in the number of chunks. A
   2,700-chunk file emits a couple of megabytes per append. Cap the
   carried chunk list at 256 entries and set
   metadata_event_chunks_truncated so subscribers that need the content
   read the entry from the filer instead of trusting the event.
   Subscribers that only track names, attributes, or chunk deltas keep
   working unchanged.

2. appendToFile uploads the chunk before committing the filer entry.
   When the commit fails — store error, cancelled context, filer
   shutting down — the chunk is stranded: no entry references it, so no
   later delete can mark it, and neither vacuum nor fsck can see it
   because both work from the entry index. The volume server keeps
   counting those bytes as live data forever. Release the chunk when
   the commit fails.

* filer: keep chunk lists in metadata events

Mount meta caches, peer filers replaying into their own store, and the
filer.sync, filer.backup and filer.meta.backup sinks all store the event's
entry as the file. Dropping the chunk list past a count left those copies
of a large file without content.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* filer: release a metadata log chunk the lookup or commit leaves unreferenced

The flush retries a failed append with a new upload, so a store outage
left one stranded chunk per retry. A failed lookup strands the chunk too.
Use DeleteUncommittedChunks, as the other write paths do: the chunk was
never part of the file, so path chunk-deletion rules do not apply.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* filer: keep a metadata log chunk the failed create stored

CreateEntry inserts a new entry before its parent folders, so a parent
failure returns an error with the entry already referencing the chunk.
Release it only when the store shows no entry, or an entry without it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* filer: count a metadata log piece the failed create stored as appended

The flush retry would append the same bytes after it, and a piece cut
inside a record then breaks replay of that segment. The retry takes the
update path, which does not create parent folders either, so returning
the error only duplicated the piece.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Martin Morez <martin@morez.org>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-06 13:19:09 -08:00
Chris LuandClaude Opus 5.5 2be4498c04 s3 lifecycle: walker deletes carry the listed identity and must target the latest version (#12452)
* s3 lifecycle: send the listed entry's identity on walker deletes

The daily walker dispatched LifecycleDelete with no ExpectedIdentity, so
the server skipped its compare-and-swap and an object rewritten between
listing and dispatch was deleted. The list adapter now records each
listed entry's identity, computed the same way as on the event path, and
the walker sends it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3 lifecycle: current-version expiration must target the latest version

The walker addresses the latest version of a versioned key by id. A
version file never changes, so its identity still matches after a newer
PUT, and the delete went ahead: a delete marker over the new version with
versioning enabled, or the new null version deleted while suspended.
LifecycleDelete now resolves the key's latest version for
EXPIRATION_DAYS/EXPIRATION_DATE requests that carry a version id and
answers NOOP_RESOLVED (VERSION_NOT_LATEST) when it is a different one.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3 lifecycle: check the latest version for unaddressed suspended expirations

With versioning suspended, an expiration without a version id reads the
plain entry, which is the null version but not necessarily the latest:
a newer real version can sit in .versions. The latest-version check now
runs in the expiration branch, by versioning state: for every suspended
request (an unaddressed one addresses the null version) and for enabled
requests that carry a version id.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3 lifecycle: test that each version sibling carries its own identity

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-06 09:39:12 -08:00
Chris LuandClaude Opus 5.5 9af736da62 volume: stop serving EC reads from quarantined shards (#12450)
* storage: skip quarantined EC shards on local reads

A needle read tried the local shard first even when its disk was
quarantined after repeated IO errors, so every read paid the failing
disk's EIO latency before falling back. Treat such a shard as not local
so the read goes to peers or reconstruction, and read it only as a last
resort when no healthy copy answers. Scrub still reads it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* volume: refuse EC shard reads from a quarantined disk

VolumeEcShardRead served peers from a shard whose disk was quarantined,
and it ignored the .ecx lookup error: a failed index read returns a
tombstone size, so the peer was told a live needle was deleted. Refuse
quarantined shards and fail the request on an index read error; a
needle missing from the index still reads, as before.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* storage: report quarantined EC shards deleted right away

Quarantining an EC volume only dropped it from later full EC heartbeats,
so the master kept routing reads to the faulty disk for up to 17
heartbeat intervals. The heartbeat that quarantines the volume now
carries its shards as deleted, once.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* storage: do not announce EC shards mounted onto a quarantined volume

MountEcShards and the shard load notifier reused a quarantined EcVolume
and announced the new shard, pointing the master back at the faulty disk
until the next full EC heartbeat. Skip the announcement, and fail the
mount so a caller moving a shard here keeps its source copy.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* seaweed-volume: count IO errors on local EC shard reads

Go's readLocalEcShardInterval feeds every local shard read into the EC
volume's IO error streak; the Rust needle read and recovery gather did
not, so a failing disk under local reads never moved toward quarantine.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* seaweed-volume: skip quarantined EC shards on local reads

Mirror the Go read path: a needle read and the recovery gather treat a
shard on a quarantined disk as not local, and read it only as a last
resort once peers and reconstruction fail. Scrub still reads it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* seaweed-volume: refuse EC shard reads from a quarantined disk

Mirror Go's VolumeEcShardRead: answer Unavailable for a shard whose disk
is quarantined, so the peer tries the next holder or reconstructs.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* seaweed-volume: report quarantined EC shards deleted right away

Mirror Go: the volume heartbeat that quarantines an EC volume carries its
shards as deleted, once, instead of leaving the master to drop them at
the next full EC heartbeat.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* seaweed-volume: fail EC shard mounts onto a quarantined volume

Mirror Go's MountEcShards: the heartbeat already leaves such a shard
unannounced, so fail the mount and let a caller moving the shard here
keep its source copy.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* storage: keep reading EC needles when the shard lookup fails

A failed or short master lookup aborted the read before any shard was
tried, so the quarantined-disk last resort never ran once the master had
dropped those shards, and local shards went unread whenever the master
was unreachable. Read on with the cached locations, as the Rust server
already does.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* ec: log a failed last-resort read of a quarantined shard

When reconstruction and the quarantined read both fail, say why the local
read failed too. Go and Rust.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* ec: drop a shard whose mount was refused on a quarantined disk

The refused mount left the new shard loaded on the quarantined volume,
where it was neither served nor announced, and the in-memory claim drew
later copies of that shard back to the same disk. Unload it again,
keeping a shard that was already mounted there. Go and Rust.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* ec: finish a mount batch before reporting a failed shard

A shard that failed to mount, now including one refused on a quarantined
disk, returned before the sidecar reload, and in Rust before the
heartbeat notify, so shards mounted earlier in the same batch kept
checksum protection off and reached the master only at the next full EC
heartbeat. Stop at the failed shard but finish the rest first. Go and
Rust.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* ec: keep a peer's deleted answer when its stream fails after it

A shard read recorded the holder's deleted answer, and a stream error
after it failed the read: Rust discarded the flag, and Go let the next
source or the reconstruction result overwrite it. With the last-resort
read of a quarantined shard, that could serve a needle a peer had
reported deleted. Once the deleted answer is in, end the read on a later
stream error instead of failing it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* seaweed-volume: use the platform media error in the quarantine heartbeat test

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* storage: rebuild EC intervals from quarantined shards when healthy ones fall short

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* seaweed-volume: rebuild EC intervals from quarantined shards when healthy ones fall short

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* volume: serve a peer's EC shard read from a healthy copy on another disk

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* seaweed-volume: serve a peer's EC shard read from a healthy copy on another disk

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* seaweed-volume: refuse a quarantined EC shard read instead of serving a sibling copy

A sibling disk's copy can have its own deletion journal, and an EC blob
delete is journaled on one local copy only, so the sibling could serve a
deleted needle.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* volume: refuse a quarantined EC shard read instead of serving a sibling copy

A sibling disk's copy can have its own deletion journal, and
VolumeEcBlobDelete journals a delete on one local copy only, so the
sibling could serve a deleted needle.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* seaweed-volume: check the disk an EC shard mounted on for quarantine

mount_ec_shard now returns the disk it mounted on, and the mount RPC
checks only that disk, like Go's MountEcShards. Before, a quarantined
sibling holding the same shard failed the mount while the healthy
disk's copy stayed mounted.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* storage: test an EC mount beside a quarantined sibling

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* seaweed-volume: check the quarantined-sibling mount test by its result

A server that remounts a shard on the disk already claiming it refuses
the mount there instead. The test now only requires that the result
matches whether the healthy disk ended up holding the shard.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* storage: check the quarantined-sibling mount test by its result

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* storage: expect the quarantined-sibling mount result from the disk the store picks

The test accepted a refused mount whenever the healthy disk stayed empty,
so a handler that refused before mounting passed too. Ask the store which
disk it picks first, then require success exactly when that is the healthy
disk.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* seaweed-volume: expect the quarantined-sibling mount result from the disk the store picks

Mirrors the Go test. add_location's cross-disk reconcile mounted shard 1 on
the healthy disk too, so the test now unmounts it there first, leaving the
quarantined sibling as the only claimant, as in Go.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-06 09:38:15 -08:00
2df7d435f2 filer.remote.sync: bound the stamp and offset writes, startup offset fixes, stamp a copy (#12447)
* filer.remote.sync: bounded filer lookups, one chunk cache per process

The filer lookups on the sync path (currentEntry, isSuperseded,
uploadCurrentEntry, liveRemoteEntry, the mount directory read) ran with no
deadline. A lookup that never answers keeps its job alive, the job keeps
its path in the MetadataProcessor conflict index, AddSyncJob waits on the
condition and the daemon stops consuming events with no log line. Each
lookup now carries a 30 s deadline, so the job fails and the processor's
stop-and-resubscribe path runs.

remoteUploadReader built a chunk cache per upload; the cache's worker
goroutine is never stopped, so each upload leaked one goroutine and its
cache (21k goroutines, ~150 MB/h under lock-file churn, OOM at the memory
limit). One chunk cache per process.

(cherry picked from commit ff77fd222772d4f3b68f34d5ee39a66684f90e78)

* filer.remote.sync: a failed mount lookup at startup never resolves to now

collectLastSyncOffset returned time.Now() when the mounted directory lookup
failed, which skips every event since the saved offset. The saved offset is
used instead; without one the daemon exits so a restart retries.

(cherry picked from commit 8fd3124f5b3d3f8c3a291263a710047b4aff81ae)

* filer.remote.sync: no upload on an unanswered supersession lookup; bounded offset read

retriedWriteFile returns the supersession lookup's error instead of
treating an unanswered lookup as "not superseded": in a replay an older
snapshot could otherwise overwrite a newer remote object. The processor
retries the event. isSuperseded keeps its boolean form for the other call
sites. GetSyncOffset's KvGet carries a 30 s deadline so a startup that falls
back to the saved offset cannot hang on it.

TestGatewayUncachedRewriteIsStillUploaded asserted the upload on a failed
lookup; it now asserts the withheld upload and the lookup error.

(cherry picked from commit 8fc2e718f7cd8322615b4e40007be903ade2dbfa)

* filer.remote.sync test: keep the slices import the other gateway tests use

* filer.remote.sync: an unanswered lookup after a failed attempt ends the retry loop; an unread offset never resolves to now minus timeAgo

After a failed upload attempt the supersession lookup's error ended in
isSuperseded's boolean and the loop could upload the snapshot again
unverified. The lookup error now ends the loop (errLookupUnanswered, not
retried) and the processor retries the event. With -timeAgo set, a failed
read of the saved offset resolved to now minus timeAgo, skipping everything
older; the daemon exits instead so a restart retries.

(cherry picked from commit 55ca2a88e01d7b556d494a404b9cc263c27b2566)

* filer.remote.sync: an unresolved start offset returns to the retry loop instead of exiting

resolveLastSyncOffset returns the lookup or offset-read error; both callers
(directory and buckets) hand it to util.RetryUntil, which retries with
backoff instead of the process exiting.

* filer.remote.sync: keep doc comments on the functions they describe

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* filer.remote.sync: isSuperseded returns the lookup error itself

Its only production callers needed the error, so the boolean wrapper was
left to the tests. The tests now assert the error on a failed lookup.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* filer.remote.sync: bound the stamp written after each upload

updateLocalEntry's UpdateEntry was the last filer call on the sync path
without a deadline. A filer that never answers the stamp keeps the job and
its path in the conflict index the same way an unanswered lookup did. The
stamp carries IF_ENTRY_EQUAL, so a retried event after a timeout either
stamps again or skips a stamp that already landed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* filer.remote.sync: an unread offset never resolves to the mount's creation time

With the offset read now bounded, a timed-out read fell through to the
directory's creation time and replayed every event already synced. The
error goes back to the startup retry loop, as with -timeAgo; a missing
offset still reads as zero without an error.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* filer.remote.sync: stamp a copy of the event entry

updateLocalEntry set the RemoteEntry and dropped the rename marker on the
caller's entry before the RPC. A transient stamp failure retries the same
event in place, which then read as already synced and skipped both the
upload and the stamp. The stub filer now applies a successful update the
way the filer does, since tests had relied on that shared object.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* filer.remote.sync: a directory not created yet starts now, as before

A gateway started before the first bucket looked up a missing buckets
directory, had no saved offset, and returned an error to the retry loop
on every attempt until a bucket appeared. A missing directory with no
saved offset has nothing to replay, so it starts now as it did before.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* filer.remote.sync: take the start time before the mount lookup

A bucket created between a not-found lookup and the end of the offset
read had its creation before a start taken afterwards, so the gateway
never created its remote bucket.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* filer.remote.sync: save the resume position when a subscription ends

A failed event ends the subscription with ErrResubscribe, and the retry
loop runs startup again, recomputing -timeAgo from a later now. The
periodic checkpoint only runs as events arrive, so a failure before the
first one could start the retry after the event it failed on. The resume
position is now saved whenever the subscription returns.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* remote_storage: bound the sync offset write like the read

The offset write runs in the subscription's checkpoint callback and, now,
when a subscription ends; a filer that never answers it stalled either.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* filer.remote.sync: say that any saved offset wins over the creation time

The creation time in seconds was compared with the offset in
milliseconds, which made any saved offset win. That is the behaviour to
keep: remote.unmount recreates the directory and then saves the offset,
usually in the same second, so a seconds comparison would start at the
creation second and replay the purge. The comparison now states it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* filer.remote.sync: keep the resume position for the retry when its write fails

The retry in the same process now resumes from the position the ended
subscription returned, as filer.backup does, so a failed offset write
followed by an empty read cannot recompute -timeAgo past a failed event.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* filer.remote.sync test: give the same-second saved offset a fractional part

With whole seconds, returning the creation second resolved to the same
time and passed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: james <james@medable.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-06 09:36:06 -08:00
206d7b1418 filer.remote.sync: bounded filer lookups, one chunk cache per process (#12441)
* filer.remote.sync: bounded filer lookups, one chunk cache per process

The filer lookups on the sync path (currentEntry, isSuperseded,
uploadCurrentEntry, liveRemoteEntry, the mount directory read) ran with no
deadline. A lookup that never answers keeps its job alive, the job keeps
its path in the MetadataProcessor conflict index, AddSyncJob waits on the
condition and the daemon stops consuming events with no log line. Each
lookup now carries a 30 s deadline, so the job fails and the processor's
stop-and-resubscribe path runs.

remoteUploadReader built a chunk cache per upload; the cache's worker
goroutine is never stopped, so each upload leaked one goroutine and its
cache (21k goroutines, ~150 MB/h under lock-file churn, OOM at the memory
limit). One chunk cache per process.

(cherry picked from commit ff77fd222772d4f3b68f34d5ee39a66684f90e78)

* filer.remote.sync: a failed mount lookup at startup never resolves to now

collectLastSyncOffset returned time.Now() when the mounted directory lookup
failed, which skips every event since the saved offset. The saved offset is
used instead; without one the daemon exits so a restart retries.

(cherry picked from commit 8fd3124f5b3d3f8c3a291263a710047b4aff81ae)

* filer.remote.sync: no upload on an unanswered supersession lookup; bounded offset read

retriedWriteFile returns the supersession lookup's error instead of
treating an unanswered lookup as "not superseded": in a replay an older
snapshot could otherwise overwrite a newer remote object. The processor
retries the event. isSuperseded keeps its boolean form for the other call
sites. GetSyncOffset's KvGet carries a 30 s deadline so a startup that falls
back to the saved offset cannot hang on it.

TestGatewayUncachedRewriteIsStillUploaded asserted the upload on a failed
lookup; it now asserts the withheld upload and the lookup error.

(cherry picked from commit 8fc2e718f7cd8322615b4e40007be903ade2dbfa)

* filer.remote.sync test: keep the slices import the other gateway tests use

* filer.remote.sync: an unanswered lookup after a failed attempt ends the retry loop; an unread offset never resolves to now minus timeAgo

After a failed upload attempt the supersession lookup's error ended in
isSuperseded's boolean and the loop could upload the snapshot again
unverified. The lookup error now ends the loop (errLookupUnanswered, not
retried) and the processor retries the event. With -timeAgo set, a failed
read of the saved offset resolved to now minus timeAgo, skipping everything
older; the daemon exits instead so a restart retries.

(cherry picked from commit 55ca2a88e01d7b556d494a404b9cc263c27b2566)

* filer.remote.sync: an unresolved start offset returns to the retry loop instead of exiting

resolveLastSyncOffset returns the lookup or offset-read error; both callers
(directory and buckets) hand it to util.RetryUntil, which retries with
backoff instead of the process exiting.

* filer.remote.sync: keep doc comments on the functions they describe

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
(cherry picked from commit 952d80c4d3)

* filer.remote.sync: isSuperseded returns the lookup error itself

Its only production callers needed the error, so the boolean wrapper was
left to the tests. The tests now assert the error on a failed lookup.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
(cherry picked from commit 51f603b5ec)

* filer.remote.sync: bound the stamp written after each upload

updateLocalEntry's UpdateEntry was the last filer call on the sync path
without a deadline. A filer that never answers the stamp keeps the job and
its path in the conflict index the same way an unanswered lookup did. The
stamp carries IF_ENTRY_EQUAL, so a retried event after a timeout either
stamps again or skips a stamp that already landed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
(cherry picked from commit c7338fb68f)

* filer.remote.sync: an unread offset never resolves to the mount's creation time

With the offset read now bounded, a timed-out read fell through to the
directory's creation time and replayed every event already synced. The
error goes back to the startup retry loop, as with -timeAgo; a missing
offset still reads as zero without an error.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
(cherry picked from commit 0a1b6c2944)

* filer.remote.sync: stamp a copy of the event entry

updateLocalEntry set the RemoteEntry and dropped the rename marker on the
caller's entry before the RPC. A transient stamp failure retries the same
event in place, which then read as already synced and skipped both the
upload and the stamp. The stub filer now applies a successful update the
way the filer does, since tests had relied on that shared object.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
(cherry picked from commit 1f332e5866)

* filer.remote.sync: a directory not created yet starts now, as before

A gateway started before the first bucket looked up a missing buckets
directory, had no saved offset, and returned an error to the retry loop
on every attempt until a bucket appeared. A missing directory with no
saved offset has nothing to replay, so it starts now as it did before.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
(cherry picked from commit c31497a9a3)

* filer.remote.sync: take the start time before the mount lookup

A bucket created between a not-found lookup and the end of the offset
read had its creation before a start taken afterwards, so the gateway
never created its remote bucket.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
(cherry picked from commit f535fca8f6)

* filer.remote.sync: save the resume position when a subscription ends

A failed event ends the subscription with ErrResubscribe, and the retry
loop runs startup again, recomputing -timeAgo from a later now. The
periodic checkpoint only runs as events arrive, so a failure before the
first one could start the retry after the event it failed on. The resume
position is now saved whenever the subscription returns.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
(cherry picked from commit 7be919ded1)

* remote_storage: bound the sync offset write like the read

The offset write runs in the subscription's checkpoint callback and, now,
when a subscription ends; a filer that never answers it stalled either.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
(cherry picked from commit f888ac6b8f)

* filer.remote.sync: say that any saved offset wins over the creation time

The creation time in seconds was compared with the offset in
milliseconds, which made any saved offset win. That is the behaviour to
keep: remote.unmount recreates the directory and then saves the offset,
usually in the same second, so a seconds comparison would start at the
creation second and replay the purge. The comparison now states it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
(cherry picked from commit a0d60ee00f)

* filer.remote.sync: keep the resume position for the retry when its write fails

The retry in the same process now resumes from the position the ended
subscription returned, as filer.backup does, so a failed offset write
followed by an empty read cannot recompute -timeAgo past a failed event.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
(cherry picked from commit 39ef551bcf)

* filer.remote.sync test: give the same-second saved offset a fractional part

With whole seconds, returning the creation second resolved to the same
time and passed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
(cherry picked from commit 4bb74382a6)

---------

Co-authored-by: james <james@medable.com>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-06 09:35:16 -08:00
Chris LuandClaude Opus 5.5 1406335301 volume: a delete cancels writes of its needle still in flight (#12445)
* volume: a delete cancels the writes of its needle still in flight

A write queued behind the volume locks, for example behind another write's
fsync on a busy disk, is not in the needle map yet. A delete of that needle
in the meantime finds nothing, and the write lands afterwards, so the needle
comes back on this copy only while the other replicas have it deleted.

Each volume now tracks its writes from entry until they return. A delete
cancels the ones for its needle, and a cancelled write that landed is
tombstoned before it returns, unless the needle was deleted or rewritten
since.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* volume: a replicate delete that finds nothing cancels the pending write

Rolling back a failed chunk upload sends DELETE ?type=replicate to every
holder, including the slow one whose write timed out on the client. That
holder has not indexed the write yet, so the delete answered 404 and the
write landed live afterwards. Cancel the needle's pending writes, then look
again in case one finished in between. A plain delete that finds nothing
still answers 404 without cancelling: it is not sent on to the replicas.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* seaweed-volume: a delete cancels the writes of its needle still in flight

Mirrors the Go volume. Volume::track_pending_write registers a write until
its guard drops, delete_needle cancels the needle's pending writes, and
settle_pending_write tombstones a cancelled write that landed unless the
needle was deleted or rewritten since.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* seaweed-volume: track uploads and cancel them on a replicate delete

An upload is tracked from the handler, and a durable one from submission
to the write queue, where the guard travels with the request: a handler
dropped while the write waits in the queue does not untrack it, and the
worker settles it after the batch. A replicate delete that finds nothing
cancels the needle's pending writes and looks again, like the Go handler.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* volume: take a cancelled write back in the lock span that publishes it

The write tombstoned itself only after it returned, with the volume locks
released. In that gap a compaction could move the record so the offset
check skipped it, a close could drop it, and an identical write sent after
the delete could match the live record and report success before the
tombstone removed it. The check now runs right after the publish, under
the same dataFileSyncLock and dataFileAccessLock hold, in the plain, fsync
and batch write paths.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* volume: a delete cancels only the writes carrying its cookie

A delete with the right needle id but the wrong cookie could cancel an
in-flight write and get it tombstoned. Each pending write now records its
cookie, and a cancel only marks the matching ones.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* seaweed-volume: take a cancelled write back in the lock span that publishes it

Mirror the Go change: settle_pending_write ran after write_needles_grouped
returned, so a compaction, a close or an identical write later in the same
batch could slip in before the tombstone. drop_cancelled_write now runs
right after the publish, under the same sync and access lock hold, in the
synced run, the unsynced run and write_needle_unsynced_or_refused.
write_pending_needles carries each write's PendingWrite down to that
point.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* seaweed-volume: a delete cancels only the writes carrying its cookie

Mirror the Go change: a delete with the right needle id and the wrong
cookie no longer cancels an in-flight write. Each tracked write keeps its
cookie next to its cancel flag.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* seaweed-volume: refuse a queued write whose volume was remounted

The write queue tracks a write on the volume found at submit, but the
worker looks the volume up again by id. After a remount, a delete reaches
the new volume and cannot cancel the write registered on the old one, so
the worker now refuses such a write instead of landing it on the new
volume.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* seaweed-volume: decode the delete's type parameter like Go's FormValue

The delete handler matched the raw query text for type=replicate, so a
percent-encoded name or value was not treated as a replica delete and did
not cancel the write it undoes. Parse the query the way the upload handler
does.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* volume: test a replica delete landing in a write's fsync window

Drive DeleteHandler while a durable write is parked in its .dat fsync,
appended but not indexed: the replica delete answers 404 and the write is
taken back when it publishes, while a plain delete leaves it live.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* volume: flush a take-back tombstone before the index row that points at it

A durable write taken back during its fsync appends the tombstone after the
.dat sync, and the .idx sync that follows made its row durable while the
record itself could still be lost, which reloads the volume read only. The
fsync and batch paths now sync the .dat again first when a take-back
appended a tombstone; a failure stops the volume taking writes like a failed
index sync.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* seaweed-volume: flush a take-back tombstone before the index row that points at it

A durable write taken back in write_synced_run appends its tombstone after
the .dat sync, so the .idx flush that follows could make the row durable
while the tombstone record itself was still in the page cache. The run now
syncs the .dat again first when a take-back appended a tombstone; a failure
stops the volume taking writes like a failed index flush.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* volume: a skip-cookie batch delete cancels the writes of the file id it names

BatchDelete with SkipCookieCheck left the needle's cookie at zero, so the
delete's cancel matched none of the in-flight writes and a retry queued
behind it could land afterwards. The non-EC delete now carries the file
id's cookie, as the Rust volume server already does; the EC path keeps
cookie 0, which DeleteEcShardNeedle reads as "skip the check".

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* volume: release a held test write before the store closes

A failed wait ended the test before release ran, so the writer stayed parked
in its fsync and the store cleanup blocked on its lock until the package
timed out.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-06 09:34:20 -08:00
f71c0fe523 fix(s3tables): route REST RenameTable and refuse unrouted S3 Tables requests (#12446)
* s3tables: route REST RenameTable and refuse unrouted S3 Tables requests

PUT /tables/<arn>/<ns>/<name>/rename had no route, so it fell through to the
/{bucket} routers and ran as a PutObject into a bucket named "tables",
answering 2xx while the table kept its name.

- Answer any s3tables-signed request no S3 Tables route takes with 404
  UnknownOperationException, before the health, domain and bucket routers.
- Route the AWS RenameTable REST call (newNamespaceName, newName, both
  optional; versionToken).
- Honor versionToken on RenameTable, checked against the snapshot the claim
  asserts so a commit between the two reads cannot slip a stale token through.

* s3tables: match the credential scope's service, not a region named s3tables

The unrouted-request fallback widens what a false match costs: an S3
client whose region is named s3tables had every request answered as an
unknown S3 Tables operation.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3tables: refuse an empty newName or newNamespaceName on REST rename

An empty field was read as omitted, so {"newNamespaceName":"other",
"newName":""} moved the table under its old name instead of failing
validation the way AWS does.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3tables: check the rename version token after the snapshot comparison

Once the separately read metadata is known to equal the claimed snapshot,
the token compares against the parsed metadata; no second decode of the
snapshot bytes is needed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3tables test: cover policy and tag changes in the rename snapshot compare

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3tables test: give each REST rename subtest its own source table

The token subtests renamed the table the first subtest produced, so run
alone they failed on a missing table instead of testing the token.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-06 09:32:13 -08:00
Chris LuandClaude Opus 5.5 c84c43f3da master: export Hashicorp raft metrics through the go-metrics global raft uses (#12448)
* master: keep the hostname out of raft metric names

go-metrics prefixes gauge names with the hostname by default, which gives
every master its own metric family, e.g. SeaweedFS_<host>_raft_peers.
Prometheus already tells masters apart by the scrape target.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* master: send Hashicorp raft metrics through the go-metrics global raft uses

hashicorp/raft and raft-boltdb report through github.com/hashicorp/go-metrics,
but the master set up its prometheus sink on the armon/go-metrics global,
so a -raftHashicorp master exported no raft metric families. The sink is now
registered on the hashicorp/go-metrics global, and armon/go-metrics is no
longer a dependency.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* grafana: chart raft commit time and leader contact

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* grafana: keep the cluster in the raft panels

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* grafana: average raft commit time and leader contact over the rate window

The go-metrics summaries keep quantiles for 10s, so a p99 scraped every
30s misses most observations. The _sum/_count rates count every sample.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* grafana: leave raft panels empty when nothing was observed

Clamping the count rate plotted 0 ms for windows without commits.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* master: let the raft metrics test use its own registry

Registering on stats.Gather made go test -count=2 fail with a duplicate
collector registration.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* master: register raft metrics before opening the raft stores

raft.NewRaft starts its run loop before returning, so registering the
sink afterwards could drop the first state and boltdb samples.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-06 09:29:42 -08:00
Chris LuandClaude Opus 5.5 7b3983f154 s3api: speed up recursive listing of branched directory trees (#12438)
* s3api: look up a trailing-slash prefix's directory instead of listing its parent

A prefix such as "account-0001/" names one directory, but the walk found it by
listing the parent with the name as a prefix. SQL stores answer that with a LIKE
that bounds the scan from below only, so in a wide bucket root the query reads
every sibling after the name: seconds for a prefix holding a few keys.

Look the directory up by name and serve it as the level's only entry. A miss,
an error or a non-directory falls back to the prefix listing.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3api: track failover receives atomically

The failover wrapper flags a received reply through a plain bool, which a
callback reading from the filer on several goroutines would race on. Use an
atomic.Bool so the callback may share its filer client.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3api: read directories ahead of a recursive listing walk

A recursive ListObjects walk enters directories one at a time, so a bucket
made of many directories holding one or two keys each costs one filer
round trip per directory, in series: seconds per 1000-key page on a store
whose reads wait on disk.

Read the first window of the directories the walk will enter on a few
workers, nearest first in walk order, and hand each window over when the
walk gets there. Every window the walk reads itself is drained first so its
subdirectories are queued before the walk descends into the first one. The
number of windows held ahead grows while directories end inside their first
window and falls back when one does not, so a page ending inside a large
directory leaves few reads wasted.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3api: drop the prefetcher's record of every directory it was handed

A page can cross any number of empty directories without spending its key
budget, so a set of every announced directory grows with the tree. A
directory is already handled when it is at or behind the walk, held or in
flight, or queued; check those, which the queue and window caps bound.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3api: end a trailing-slash prefix listing at the file its lookup finds

A file named by the prefix lists nothing under it, the same as the
sibling listing it fell back to, which threw the lookup's answer away
and paid the parent scan anyway.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3api: read a walk's own listing ahead instead of draining it

Draining the window before the walk took its first entry made a page
wait for the end of the listing, which a SQL store answering a name
prefix reaches only after scanning the rest of the directory; a page
that fills before then used to return at once. A background reader now
hands the window's directories to the prefetcher as they arrive while
the walk takes each entry, holding at most 1 MiB of entries ahead.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3api: bound the bytes a prefetched listing window holds

Entries with long chunk lists or inline content made the up to 64
held windows of 32 entries an unbounded amount of memory per listing.
A window now stops at 128 KiB of entries and the walk lists the rest
of that directory from the window's last entry.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3api: test recursive listings whose streams break partway

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-06 09:28:41 -08:00
Chris LuandClaude Opus 5.5 3e64326f2b s3: require their own actions to change a bucket's CORS or versioning (#12433)
* s3: require their own actions to change a bucket's CORS or versioning

PUT/DELETE ?cors and PUT ?versioning were authorized as Write, so an
identity granted only object writes on a bucket could replace its CORS
rules or suspend versioning. They now need PutBucketCors,
DeleteBucketCors and PutBucketVersioning (or Admin), and inline policies
granting s3:PutBucketCors, s3:DeleteBucketCors or s3:PutBucketVersioning
map to those actions instead of Write.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3: test that object Write cannot change bucket CORS or versioning

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* admin: offer the bucket CORS and versioning permissions

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3: test each bucket config action with its own identity

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-06 09:12:00 -08:00
Chris LuandClaude Opus 5.5 3a579ece3b s3: commit the delete marker in the owner's finalize transaction (#12432)
* s3: commit the delete marker in the owner's finalize transaction

createDeleteMarker wrote the marker with mkFile to the gateway's current
filer and then sent RECOMPUTE_LATEST to the object's ring owner. When the
two differ, the owner scans .versions before peer metadata sync delivers
the marker, re-derives the pointer to the previous version, and the
marker stays non-current on every filer.

routedVersionedFinalize now takes an optional version entry and PUTs it
in the same ObjectTransaction ahead of the recompute, the way PutObject
and CompleteMultipartUpload already do. The copy destination keeps
passing nil: its version file is already written on the owner.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3: test a versioned delete when the gateway's filer is not the owner

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3: assert the delete marker and the pointer flip share one transaction

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-06 09:10:10 -08:00
a8766f5177 s3api: apply copy bucket-default SSE after the ACL re-auth (#12436)
* s3api: apply copy bucket-default SSE after the ACL re-auth

CopyObject without its own x-amz-server-side-encryption header takes the
destination bucket's default encryption, which CopyObjectHandler writes
into the request headers. It did so before preparePutObjectACL, which
re-authenticates a request that carries an ACL (x-amz-acl or a grant
header) for s3:PutObjectAcl. verifySignedHeadersCoverage then found the
synthesized x-amz-server-side-encryption* headers outside SignedHeaders
and rejected the copy with SignatureDoesNotMatch.

Move the bucket-default injection after the ACL check, so the
re-authentication sees the request as the client signed it. The
destination object still gets the bucket default encryption.

* s3api: assert the copy's KMS key and stored ciphertext

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Stefan Scharlott <stefans@Stefans-MacBook-Pro.local>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-06 09:08:44 -08:00
d127be610f admin: re-send the master vacuum disable every poll so it survives leader changes (#12430)
* admin: re-send the master vacuum disable every poll so it survives leader changes

The admin disabled the master's scheduled vacuum only when a vacuum plugin
worker connected or disconnected. The master keeps that flag in memory and it
is not replicated through raft, so after a leader change or a leader restart
the new leader ran its own vacuum next to the plugin worker. When both picked
the same volume the master's commit failed ("compact swap aborted: missing
.cpd/.cpx") and it dropped that replica, leaving the volume under-replicated.

Send the desired state on every poll instead, and log on the master only
when the flag actually changes.

* admin: shorten the vacuum sync comments

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Kevin <kevin@leetify.com>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-06 09:07:50 -08:00
github-actions[bot] 0dd33f5f0f docs: regenerate star history chart 2026-11-06 00:33:08 +00:00
Chris LuandClaude Opus 5.5 878979af70 filer: hide write controls in the UI on the read-only port (#12429)
* filer: hide write controls in the UI on the read-only port

The read-only port accepts only GET and HEAD, but its directory page
offered New Folder, Upload, drag-and-drop, rename and delete, each of
which failed with 405. Requests that come through the read-only handler
now render the page without those controls or the upload script.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* filer: test the directory page controls on each port

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* filer: match the rendered row buttons in the port test

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-05 10:11:52 -08:00
e44c16802d weed/server: verify data shards against bitrot sidecar before ec decode (#12422) (#12427)
* weed/server: verify data shards against bitrot sidecar before ec decode (#12422)

* ec.decode: verify data shards against the mounted volume's bitrot protection

The decode looked up only the generation-0 .ecsum beside the data base and
each location's data directory, so it missed a sidecar kept with the index
and ignored the generation the volume actually serves.

Take the protection the volume's per-disk runtimes resolved instead (On, else
Invalid, else Off), and test a corrupt block past the first.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* ec.decode: check for live entries before verifying the shards

A volume whose needles are all deleted gets its shards purged on the
no-live-entries status; a failing checksum must not keep it around.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* seaweed-volume: verify data shards against the bitrot protection before a decode

Mirrors the Go volume server: the decode takes the protection the volume's
per-disk runtimes resolved and refuses a data shard that fails its .ecsum
checksum, after the no-live-entries check.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test: a decode refuses a data shard that fails its .ecsum checksum

Runs against both volume server implementations: the refused decode keeps
every shard and the sidecar, and the needle stays readable through EC.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* ec.decode: reload the bitrot sidecar before verifying the shards

A rebuild's backfill writes the sidecar without reloading the mounted
runtimes, which then still report no protection; resolve it again across
the EC metadata directories, as a shard mount does.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* seaweed-volume: reload the bitrot sidecar before verifying the shards

Mirrors the Go volume server: the decode job resolves the sidecar again on
every runtime of the volume before choosing its protection.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test: a decode finds a sidecar written after the shards were mounted

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test: read through the EC shards after the refused decode

The source volume stayed mounted, so the final read never touched the
shards it is meant to check.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* ec.decode: refuse when a sidecar reload reports another layout

ReloadBitrotSidecar logged and dropped the layout-mismatch error, so the
runtime stayed Off and the decode skipped the data-shard check. It now
returns that error and the decode refuses on it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* seaweed-volume: refuse a decode when a sidecar reload reports another layout

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test: a decode refuses a late sidecar that records another layout

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* ec.decode: refuse when an unmount left no runtime to resolve the sidecar

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* seaweed-volume: refuse a decode when an unmount left no runtime to resolve the sidecar

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-05 10:11:09 -08:00
80f32bc074 weed/command: skip remote bucket delete when bucket is unmounted (#12418) (#12425)
* weed/command: skip remote bucket delete when bucket is unmounted

* remote_storage: count deleting an already-missing bucket as done

filer.remote.gateway can see the same bucket delete more than once, for
example once per filer when several filers peer their metadata. A copy
that still finds the mount mapping calls DeleteBucket again, and S3
(NoSuchBucket), Azure (ContainerNotFound, ContainerBeingDeleted) and GCS
(404) all failed it, which held the sync offset on the delete. A
missing bucket now counts as deleted, the way deleting a missing object
already does.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* filer.remote.gateway: keep the mapping table writable after mount.mapping is deleted

Deleting mount.mapping replaced the gateway's mappings with a table whose
map was nil, so the next bucket create panicked writing its mount into
it. Reset to an empty map instead.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* filer.remote.gateway: count a delete of an absent object as done

filer.remote.sync already treats ErrRemoteObjectNotFound from DeleteFile
as deleted, since GCS reports a missing object that way where S3 and
Azure succeed. The gateway returned it, so a delete event seen twice, as
it is when several filers peer their metadata, held the sync offset on
GCS. Use the same deleteRemoteFile.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* filer.remote.gateway: drop cached remote clients after the new gateway tests

GetRemoteStorage caches clients by configuration, so a repeated run
(-count=2) got the first run's recorder back and saw no calls.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* filer.remote.gateway: keep the default storage across mount.mapping changes

The default storage for new buckets is derived at startup and kept only
in the gateway's copy of the mappings; mount.mapping never stores it.
Replacing or deleting that file dropped it, so a later bucket create
with no -createBucketAt skipped the remote bucket.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-05 05:06:41 -08:00
Chris LuandClaude Opus 5.5 c971252871 admin: make the cluster lock's fairness window configurable (#12426)
* admin: make the cluster lock's fairness window configurable

Scheduled jobs hold the cluster admin lock for their whole run. Once the
lock had been held for a minute, every new job waited until all running
jobs had released it, so with hour-long EC jobs dispatch ran in waves and
most workers sat idle between them.

-adminLock.fairnessWindow keeps the one-minute default; 0 lets new jobs
join the shared hold, and shell clients get the lock when the scheduler
next drains.

Claude-Session: https://claude.ai/code/session_011UpvafcfUuC2mLxNAXLQJi

* admin: wait for the blocked acquire before disabling the fairness window

The wake-up test slept 50 ms before SetFairnessWindow(0), so a late
acquire that started after the change joined without exercising the
Broadcast. Wait until it is registered as a waiter instead.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-05 05:02:41 -08:00
8c8d0f1cdf weed/shell: resolve collection from topology when ec.decode is given only volumeId (#12424)
* weed/shell: resolve collection from topology when ec.decode is given only volumeId

When ec.decode is invoked with -volumeId for a volume that belongs to a named collection without explicitly specifying -collection, DoEcDecode was called with an empty collection string. Target volume servers then rejected VolumeEcShardsCopy with 'CopyFile not found ec volume id <vid>' because they looked under the default collection.

This commit introduces FindEcVolumeCollection to discover the collection name directly from topology metadata (checking EC shard info and regular volume info), ensuring DoEcDecode receives the accurate collection.

* ec.decode: resolve every batch volume's collection from the topology

The batch passed -collection through unless it looked like a regex, so a
comma list (a,b), a {n} quantifier, ALL or _default_ reached the volume
servers as a collection name and the copy found no shards. The selector
only picks the volumes; each one is decoded in its own collection.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* ec.decode: take the collection only from the volume's ec shards

The lookup returned the first record per disk, so a regular volume on an
earlier node answered before the ec shards on a later one. A decode needs
the shards anyway, so their records are the only ones to read.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* ec.decode: leave the -volumeId lookup to DoEcDecode

DoEcDecode already resolves an empty collection from the topology.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-05 05:01:25 -08:00
120d17a092 filer.remote.sync: stop a touched file from being restamped forever (#12419)
* filer.remote.sync: stop a touched file from being restamped forever

A metadata-only update that moves the mtime past RemoteMtime (touch, or
an O_TRUNC open of an empty lock file) goes through UpdateFileMetadata,
which changes nothing on the remote, and the entry is stamped with the
same RemoteEntry and a new LastLocalSyncTsNs. That stamp is an update
event of its own. shouldSendToRemote only looks at LastLocalSyncTsNs
when RemoteMtime and the mtime share a second, so it still reads the
entry as dirty, and the event is stamped again. Nothing is uploaded,
but the daemon keeps rewriting the entry as fast as it reads its own
events back.

Skip the restamp when the backend left the object alone and the event's
entry already carries a stamp taken after its mtime. The touch itself is
still stamped once.

* filer.remote.sync: log the skipped restamp

The other skips in processUpdateEvent log at V(2); this one did not, so
with -v=2 a touched file showed an "update meta" line with nothing after
it.

* filer.remote.sync: skip the restamp of a stamp's own event, whatever the clocks

The skip compared the entry's stamp with its mtime, so an mtime ahead of
the sync's clock (a skewed writer, touch -d) stayed past every stamp and
kept the loop going until the clock caught up. An event that changed only
the RemoteEntry is the sync's own stamp, which needs no other.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* filer.remote.sync: ignore the filer's commit timestamp when spotting a stamp

The filer gives every stored update a new CommitTsNs, so a stamp's event
never compared equal outside the RemoteEntry and a touched file kept being
restamped against a real filer. The test now assigns one per update.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-05 04:08:28 -08:00
23598afb31 volume: bound each replica forward so a silent peer cannot hold a write (#12417)
* volume: bound replica forwards so a silent peer cannot hold a write

The Rust volume server forwards each replicated write to its peers with the
shared outgoing reqwest client, which had no connect or request timeout.
When a peer is silently partitioned (packets dropped, no reset) the forward
waits until TCP gives up, minutes later, and the write waits with it. A
chaos test (replication 001, one volume server partitioned) saw every write
on a volume with a copy there block.

- Replica forwards go through replica_request, bounded at
  REPLICA_REQUEST_TIMEOUT = 10 s from connect to the response.
- Both outgoing clients start from outgoing_client_builder, which adds a
  5 s connect timeout, so new connections to a peer that never answers
  fail fast. The shared client gets no overall request timeout, since it
  also serves lookups and other transfers.

Tests: a replica that receives the forward and never answers fails the
whole replicated write at the bound, naming that replica; a silent peer
fails the request builder at the bound on a paused clock; a connect that
stalls in the resolver fails at the 5 s connect bound.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* volume: give a replica forward more time for a larger body

The 10 s replica forward bound runs from connect to the end of the
response, so it also covers sending the body. A write near the 256 MB file
size limit needs about 26 MB/s to reach its replica in 10 s, so on a link
that is slower or busy the forward failed and the write returned 500 even
though the peer was answering.

A forward now gets REPLICA_REQUEST_TIMEOUT plus one second per MiB of
body. A bodiless forward keeps the 10 s bound (a 1 MiB write gets 11 s).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* volume: bound each Go replica forward like the Rust one

The Go volume server forwards a replicated write with a 10 s dial bound and
the transport's 30 s response header timeout. A replica that takes the
forward and never answers held the write for 30 s, and a body larger than
the socket buffers to a silently partitioned peer was held until TCP gave
up, since nothing bounds sending the body.

Each forward now runs under replicaRequestBound: 10 s plus one second per
MiB of body, the same bound the Rust volume server puts on its forwards.

Test: a replica that receives the forward and never answers fails the
write at the bound (300 ms in the test); without it the write fails at
30 s on the response header timeout.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* volume: keep proxy variables out of the stalled-connect test

reqwest reads HTTP_PROXY and ALL_PROXY by default, so with a proxy set the
test's request went to the proxy instead of the stalled resolver.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* volume: count a replica forward that hit its bound as a timeout

DistributedOperationResult joined the replica errors into a plain string,
and the uploader reports a cancelled request without wrapping the context
error, so a forward cut off at replicaRequestBound was counted under
server_error in the replication failure metric, as was a response header
timeout before it.

The joined error now unwraps to each replica's error, and a forward that
ran out of its bound reports context.DeadlineExceeded.

Test: the silent replica write raises the timeout failure count by one;
with the flattened join it stays put.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* volume: test that a large Go replica forward gets its body's bound

A replica answers after the bodiless bound but within the bound for a
1 MiB body, so the write fails if replicateTo drops the body scaling.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* volume: run the Rust whole-path replica test with a 1 MiB body

The forward then has to wait the bound for its body, so the test fails if
do_replicated_request stops passing the body length to the bound.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: pingqiu <pingqiu@gmail.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-05 04:05:33 -08:00
Chris LuandClaude Opus 5.5 aab947b65e s3api: answer success when a failed completion already committed (#12416)
* s3api: name the check that a completion already committed

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3api: answer success when a failed completion already committed

A finalize can fail after its commit: the filer stored the object and the
call still reported an error. The client then got a 500 with an unknown
outcome. When the failure may have left this attempt's object behind,
check the key the way a retry would and answer success with the
committed result when it holds this upload's object.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3api: keep part data when the commit found is another attempt's

The object carrying this upload ID can be a concurrent attempt's commit of
the same upload, and its part selection can include entries this attempt
counts as unused. Free those parts' data only when the committed chunks
are this attempt's own; otherwise drop their metadata only.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3api: test the committed-completion read on versioned buckets

Cover the latest-version read (version id and checksum in the answer),
a suspended bucket whose delete marker is still current, and that the
upload directory is removed without its data.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3api: test versioned recovery through the completion handler

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-05 03:38:52 -08:00
Chris LuandClaude Opus 5.5 fe1cf5263f volume: bound each FetchAndWriteNeedle replica write (#12415)
* seaweed-volume: bound each FetchAndWriteNeedle replica write

A replica that accepted the connection and then went silent held the
request until TCP gave up, because the shared outgoing client has no
request timeout. Each replica write now fails after 30 s, from connect
to the response status, matching the Go volume server.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* volume: bound each FetchAndWriteNeedle replica write

The default replica path uploads through the guarded peer client, which
bounds only the dial and has no response header timeout, and the filer
calls FetchAndWriteNeedle with no deadline. A replica that accepted the
connection and went silent held the request for minutes. Each replica
write, retries included, now has 30 s.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* seaweed-volume: require the silent-peer test to fail at the deadline

A connection error that came back early also passed, since the test
only checked an upper bound. It now also requires the paused clock to
have reached FETCH_AND_WRITE_REPLICA_TIMEOUT.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* seaweed-volume: grow the FetchAndWriteNeedle replica bound with the chunk

The reqwest timeout includes sending the body, and a remote-cache chunk
is a thousandth of the object once it passes the 1000-chunk cap, so a
fixed 30 s would fail a large chunk on a link under ~35 MB/s. The bound
is now 30 s plus 1 s per MiB.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* volume: grow the FetchAndWriteNeedle replica bound with the chunk

The context deadline includes sending the body, so a fixed 30 s would
fail a large remote-cache chunk on a slow link. The bound is now 30 s
plus 1 s per MiB, matching the Rust volume server.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-05 03:37:49 -08:00
Chris LuandClaude Opus 5.5 247e010865 volume: redial a master that just dropped a registered stream after a second (#12420)
* volume: redial a master that just dropped a registered stream after a second

A registered volume server that loses its master with no peer naming a
leader waited a full pulse before redialing. A restarted master is
usually back within a second, so its volumes stayed unregistered, and
uncached lookups kept failing, for most of a pulse.

For one pulse after a stream a leader had answered ends, redial on the
one-second cap. A master still down after that is redialed at the pulse
pace, so redials never come later than before.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* seaweed-volume: redial a master that just dropped a registered stream after a second

Mirror of the Go heartbeat change. The loop keeps when the last stream a
leader had answered ended, which also stands in for the
confirmed-leader-once flag (None until first registration), and redials
on the one-second cap for a pulse after that.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* seaweed-volume: test that a dropped registered stream records the loss

A fake master confirms the stream and then ends it. The server starts as
registered long ago, so only a recorded loss makes the redial come after
a second instead of a pulse.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* volume: record a lost stream only after its receiver has exited

doHeartbeatWithRetry can return on a send error while the receiver is
still handling the leader's first reply, so the deferred recorder could
read the registered flag too early and keep an old loss time. Cancel the
stream and wait for the receiver before reading it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test: require the prompt heartbeat redial to wait out its second

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* seaweed-volume: fail the down master's dials at once in the redial test

Windows takes about two seconds to refuse a connection to a closed
loopback port, so the stream dial and the leader lookup each held the
prompt case past its two-second bound. A listener that closes every
connection fails both dials immediately on every platform.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-05 03:27:45 -08:00
Chris LuandClaude Opus 5.5 8cc3a8ae21 ec.rebuild: repair shards that fail a checksum scrub in complete volumes (#12410)
* ec.rebuild: rebuild shards that fail a checksum scrub, also in complete volumes

ec.rebuild skipped every volume with all shards present, so a shard whose
bytes no longer match the .ecsum sidecar stayed in place until some other
shard went missing. With -checksum, ec.rebuild first scrubs each node's
shards in checksum mode and takes the failing copies out of the plan; a
shard left without a good copy counts as missing and is regenerated.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* ec.rebuild: remove the copies that failed the checksum once a good copy serves

A shard regenerated on the rebuilder left the failing copy mounted on its
old holder, so the volume kept a corrupt duplicate that a later dedup could
keep instead of the good one. Remove each failing copy once the rebuilt
shard is mounted, and right away when another holder already serves a copy
that passed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* ec.rebuild: test the checksum scrub against fake volume servers

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* ec.rebuild: check the lock before removing a corrupt copy

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* ec.rebuild: print the checksum scrub details

A malformed sidecar or a wholesale mismatch comes back only in the scrub
details, with no broken shard, so the volume would look verified.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* ec.rebuild: fail the run when a corrupt duplicate cannot be removed

For a volume whose only problem is a corrupt duplicate, the removal is the
whole repair, so its failure has to reach the caller.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* ec.rebuild: test duplicate removal failures and a lost lock

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* ec.rebuild: fail the run when a node's checksum scrub fails

The node's copies stay in the plan unverified, so a corrupt shard there is
left in place; the rest of the cluster is still repaired.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* ec.rebuild: preview the corrupt-copy removals in dry-run

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* ec.rebuild: test a failed checksum scrub

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-05 02:08:22 -08:00
Chris LuandClaude Opus 5.5 fcf62db8d0 seaweed-volume: stage optional EC sidecar copies before replacing (#12403)
* seaweed-volume: stage optional EC sidecar copies before replacing

VolumeEcShardsCopy opened the destination .vif/.ecsum with File::create
and then streamed into it. A source without the file left a 0-byte .vif,
wiped an existing .ecsum (turning bitrot protection off on that holder),
and a failed stream removed the destination's existing copy.

Stream optional copies into <file>.copying and rename into place only
once the source sent bytes, as the Go volume server's writeToFile does
for ignore_source_file_not_found.

Claude-Session: https://claude.ai/code/session_011UpvafcfUuC2mLxNAXLQJi

* seaweed-volume: keep the drain doc comment on drain_copy_stream_to_file

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* seaweed-volume: stage the .vif copied during missing-index recovery

fetch_ec_index_from_one_peer drained the optional .vif straight into the
local file, so a peer without one left a 0-byte .vif (losing a local EC
ratio) and a failed stream left a partial one. Stage it and rename only once
the peer sent bytes, as Go's recovery does through writeToFile.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* seaweed-volume: test optional copies against a stream that fails mid-transfer

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* seaweed-volume: remove a staged optional copy when the RPC is cancelled

A cancelled CopyFile handler drops its future at the stream await, so the
cleanup after it never ran and the .copying file stayed behind. Go's
writeToFile runs to completion and removes it. A guard now owns the
staging cleanup and is disarmed only once the rename commits.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* seaweed-volume: test optional copies dropped mid-stream

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* seaweed-volume: cancel the staged-copy test only once the staging file exists

tokio's File::create runs on the blocking pool, so a fixed timeout could
drop the .ecsum copy before .copying existed and pass without exercising
the cleanup guard.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-05 02:04:49 -08:00
Chris LuandClaude Opus 5.5 85232f7414 seaweed-volume: run the preallocating max-volume test on a fake disk (#12423)
* seaweed-volume: run the preallocating max-volume test on a fake disk

The test sized the volume limit from the real free space but the store
subtracts the 1% floor first, so on a disk more than 95% full the second
slot no longer fit and the count came out 1.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* seaweed-volume: start the preallocating max-volume test above the expected count

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-05 02:03:07 -08:00
Chris LuandClaude Opus 5.5 b74111ea2f s3vectors test: skip a query lease tag released between list and HEAD (#12409)
A query's lease tag is deleted by a background task after the query
answers, so the encryption helper could list it and then get a 404 on
HEAD. Skip only that case; any other missing object still fails.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-05 02:01:18 -08:00
Chris LuandClaude Fable 5.1 48df16ea0d weed-worker: keep a vector index covering as its table grows (ANN phase 3) (#12414)
* seaweed-vector: build and extend the vector index

weed_lance_worker::vector_index builds an IVF_FLAT index in the metric the
table was created with, sizes partitions from the row count, and extends an
index with a new segment or a retrain; the core crate re-exports it and adds
the probes a query spends. Tests cover coverage after build, append, retrain
and compaction.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* weed-vector-cli: index build and index optimize

Logs go to stderr so stdout stays the JSON a caller parses.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* test/s3vectors: PERF_INDEX builds the index before the timed queries

The cluster fixture hands out its catalog URL, the Makefile builds
weed-vector-cli, and the perf suite builds the index after ingest and
reports the build time and segment count.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* seaweed-vector: probe a tenth of the index's partitions, more under a filter

A query that names no probe count gets one from the index's partition
count. Lance searches at least that many partitions, and under a prefilter
keeps going until it has k rows; without one it stops there.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* weed-vector: --nprobes overrides the probe policy

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* seaweed-vector: test that a build takes the table's metric

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* test/s3vectors: make clean removes weed-vector-cli

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* seaweed-vector: measure recall with vectors the table does not hold

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* weed-vector: keep the read cache's doc comment on with_read_cache

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* weed-worker: lance_vector_index builds the index of an S3 Vectors table past a row threshold

Detection keeps the tables a vector bucket created, with at least min_rows
rows (50,000 by default) and no index on their vector column. Execution
builds the IVF_FLAT index in the table's metric with sqrt(rows) partitions.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* seaweed-vector: optimize only the indices on the vector column

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* seaweed-vector: resolve the query metric with index::distance_metric

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* seaweed-vector: test the L2 metric under the name CreateIndex stores

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test/s3vectors: build the index through the admin server and the Lance worker

VECTOR_WORKER=1 adds weed admin and weed-worker to the cluster. The new
test sets the job's threshold, runs it through the plugin API, checks
coverage and recall, and that a row written after the build is found.
CI gains the worker leg for both volume servers.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* weed-worker: read an index's partitions, and when a table has outgrown them

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* seaweed-vector: probes from the shared partition reader, partitions in coverage

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* weed-worker: a vector index reindexes at a tenth of its rows and retrains once the table outgrows it

lance_optimize_indices proposes a vector table once its uncovered rows pass
min(the absolute budget, a tenth of the indexed rows), and trains the index
again instead of extending it when sqrt(rows) has reached twice its
partitions. The end-to-end test covers both.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* weed-worker: open a table inside a caller's Lance session

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* weed-vector: keep opened indices between queries in one session

Every open used a fresh Lance session, so the index and metadata caches
died with the request and each query loaded the IVF index again.
--index-cache-mb sizes the shared cache, 512 MiB by default.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* test/s3vectors: PERF_CLUSTERS draws the perf suite's vectors around shared centres

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* test/s3vectors: the index test draws embedding-like vectors

Independent Gaussian vectors have no locality for an IVF index to use, so
recall through one only tracks the fraction of partitions probed; points
around shared centres behave like embeddings. The test must also run
before the one that stops the engine.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* test/s3vectors: the optimize and retrain steps draw embedding-like vectors

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* seaweed-vector: test that compaction drops the deleted key from tied hits

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* seaweed-vector: test that an append-only optimize serves the new row

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test/s3vectors: record PERF_NOISE in the perf report config

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* seaweed-vector: a filtered query widens to four times the probes, not without bound

With no upper bound Lance prepares every partition for the late search
even when the minimum probes already hold k rows, and a filtered query
through the index cost as much as the exact scan. A query that names its
own probe count gets exactly that count.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* test/s3vectors: report recall through an index without asserting it

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test/s3vectors: count a hit tying the k-th best score toward recall

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* admin: lance_vector_index runs in the Lance lane

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* weed-worker: check a table's index before counting its rows

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* test/s3vectors: make build and clean cover the worker leg's binaries

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* weed-worker: a retrain rebuilds the index with partitions for the row count

Lance's OptimizeOptions::retrain does not size the partitions: it left a
6,000-row index with one. The retrain goes through the builder with
sqrt(rows) partitions, for the job and for weed-vector-cli index optimize
--retrain alike. The end-to-end test no longer expects a merge to keep the
partition count.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* test/s3vectors: allow float32 rounding at the recall cutoff

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test/s3vectors: count a repeated hit key once toward recall

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test/s3vectors: credit cutoff ties only up to the slots the top-k leaves

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test/s3vectors: assert every filtered hit passes the filter, indexed or not

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* weed-worker: coverage over an index's names, one name extended, a merge at the delta limit

A sharded build leaves one name per shard, each missing the others'
fragments, so coverage is the union of every index on the column and an
optimize gives new rows to the name with the fewest deltas only. At
MAX_DELTAS the policy passes OptimizeOptions::merge(deltas); Lance's
default options may append instead of merging. A retrain keeps an index's
own metric and rebuilds only IVF_FLAT indices.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* weed-worker: the optimize job measures and extends a vector table's indices together

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* seaweed-vector: tests for bounded deltas, disjoint shards and a retrain's metric

Nine append-and-optimize cycles give delta counts 2 to 8 and then 1; two
names on disjoint fragments stay disjoint through an optimize; a cosine
index on a table without metric metadata is still cosine after a retrain.
The end-to-end test expects the appended delta after the first optimize.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* weed-vector: forget the shared session when a table was dropped and created again

The session keys index lists and transactions by dataset version, and a
table created again at the same location restarts its versions. An open
that finds an older version than the newest seen at that URI, or the same
version committed later, replaces the session and opens again.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* weed-vector: test an open after another engine created the table again

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* weed-worker: retrain only a single IVF_FLAT vector index name

The job compared each shard name's partitions with the whole table's
rows, so a sharded index was always due, and a non-IVF_FLAT index made
optimize bail on every run before the new rows were indexed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* weed-worker: refuse to retrain a vector index sharded over several names

A rebuild per name over the whole table left every shard covering every
fragment, so a query searched each row once per shard.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* seaweed-vector: test which vector indices a retrain is due for

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* weed-vector: replace the session in the same lock that records a recreated table

A second open of the recreated table could see the new version recorded
and the old session still shared, and keep the dropped table's index list.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* weed-worker: skip a vector table whose rows cannot be counted

Detection returned on the first table whose count_rows failed, so one
unreadable table kept every other table from being proposed. It now skips
that table like an open or index failure.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* grafana: panel for vector indexes the Lance worker built

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* seaweed-vector: size default probes from the largest index on the column

A sharded build gives the column one index name per shard. The first
name alone could be a small shard, and its probe count would then be
spent on a larger one.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* seaweed-vector: skip the probe lookup when a query is given no segments

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* weed-vector: open every table in its own Lance session again

One session shared by every open keeps a table's index loaded between
queries, but Lance keys what it caches by location and version, so a
table dropped and created again at the same location could be served
the old table's index list. Telling the two apart from version and
commit time needs a map that only grows, a retry loop and clocks that
agree across engines. A private session per open, as before, has no
such state. Measured at 50k x 384 through an index, eight clients: 164
queries/s with the shared session, 137 without; one client is unchanged.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* weed-vector: reject --nprobes 0

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* seaweed-vector: read the partition count from the opened index

Lance's index statistics took 3.6 ms a query on a 200k-row table with
447 partitions. Opening the index and asking it takes 0.4 ms, and the
scan that follows finds the index already open.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* ci: give the S3 Vectors worker legs the vendored protoc

The worker workspace has no protoc of its own for lance's build
scripts, so the worker legs failed at their first build.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* weed-worker: give a vector index build an hour before the admin times it out

The job set no execution timeout, so the admin's default of 90 seconds
applied, and a build that trains on a large table would be reported as
failed and retried while it was still running.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* weed-worker: lance_vector_index proposes at most the admin's max_results

Every qualifying table went into one message however many there were.
The detector stops at the limit the request carries and sets has_more,
as the Go detectors do.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* weed-worker: a vector index job re-checks the table it was proposed for

A table emptied since detection completes without a build, and one
without a vector column fails instead of reporting it already indexed.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* weed-worker: lance_optimize_indices gets a build's execution timeout

A retrain rebuilds the index over the whole table, which the admin's
default of 90 seconds would cut off on a large one.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* weed-worker: a vector index run window as long as one build

An attempt is capped at the end of the job type's run window plus the
drain grace, and the default window is 30 minutes, so a build could be
cut off before its own hour was up. The window is an hour too, as the
Go iceberg handler sets both.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* weed-worker: a lance_optimize_indices run window as long as one retrain

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* weed-worker: vector index proposals declare a build's runtime

A run started by hand while the job type's schedule is off takes no
timeout from the descriptor, only the admin's 90 seconds. The admin
gives a job at least the runtime its proposal declares, so the build
and the optimize proposals of a vector table declare the hour.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* seaweed-vector: coverage reports no partitions for an index of unknown coverage

Reading the partition count goes through Lance's index statistics,
which fail on an index that records no fragment bitmap, and the whole
coverage report failed with them.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* weed-worker: an index that records no fragments is not read as uncovered

Its rows counted as uncovered, so detection proposed the table and the
execution then failed in Lance's statistics. Coverage fails for such an
index instead, and detection skips the table with a warning.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* seaweed-vector: coverage reads partition counts from the opened index

Lance's index statistics rewrite the manifest of a table whose
fragments carry no row counts, which a coverage report must not do and
cannot do with read-only credentials. The report asks each opened
segment instead, as the probe lookup does, and the statistics-based
reader has no caller left.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

---------

Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
2026-11-04 23:30:27 -08:00
Chris LuandClaude Fable 5.1 e465880ee8 weed-worker: build the vector index of S3 Vectors tables past a row threshold (ANN phase 2) (#12413)
* seaweed-vector: build and extend the vector index

weed_lance_worker::vector_index builds an IVF_FLAT index in the metric the
table was created with, sizes partitions from the row count, and extends an
index with a new segment or a retrain; the core crate re-exports it and adds
the probes a query spends. Tests cover coverage after build, append, retrain
and compaction.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* weed-vector-cli: index build and index optimize

Logs go to stderr so stdout stays the JSON a caller parses.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* test/s3vectors: PERF_INDEX builds the index before the timed queries

The cluster fixture hands out its catalog URL, the Makefile builds
weed-vector-cli, and the perf suite builds the index after ingest and
reports the build time and segment count.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* seaweed-vector: probe a tenth of the index's partitions, more under a filter

A query that names no probe count gets one from the index's partition
count. Lance searches at least that many partitions, and under a prefilter
keeps going until it has k rows; without one it stops there.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* weed-vector: --nprobes overrides the probe policy

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* seaweed-vector: test that a build takes the table's metric

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* test/s3vectors: make clean removes weed-vector-cli

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* seaweed-vector: measure recall with vectors the table does not hold

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* weed-vector: keep the read cache's doc comment on with_read_cache

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* weed-worker: lance_vector_index builds the index of an S3 Vectors table past a row threshold

Detection keeps the tables a vector bucket created, with at least min_rows
rows (50,000 by default) and no index on their vector column. Execution
builds the IVF_FLAT index in the table's metric with sqrt(rows) partitions.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* seaweed-vector: optimize only the indices on the vector column

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* seaweed-vector: resolve the query metric with index::distance_metric

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* seaweed-vector: test the L2 metric under the name CreateIndex stores

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test/s3vectors: build the index through the admin server and the Lance worker

VECTOR_WORKER=1 adds weed admin and weed-worker to the cluster. The new
test sets the job's threshold, runs it through the plugin API, checks
coverage and recall, and that a row written after the build is found.
CI gains the worker leg for both volume servers.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* weed-worker: open a table inside a caller's Lance session

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* weed-vector: keep opened indices between queries in one session

Every open used a fresh Lance session, so the index and metadata caches
died with the request and each query loaded the IVF index again.
--index-cache-mb sizes the shared cache, 512 MiB by default.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* test/s3vectors: PERF_CLUSTERS draws the perf suite's vectors around shared centres

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* test/s3vectors: the index test draws embedding-like vectors

Independent Gaussian vectors have no locality for an IVF index to use, so
recall through one only tracks the fraction of partitions probed; points
around shared centres behave like embeddings. The test must also run
before the one that stops the engine.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* seaweed-vector: test that compaction drops the deleted key from tied hits

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* seaweed-vector: test that an append-only optimize serves the new row

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test/s3vectors: record PERF_NOISE in the perf report config

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* seaweed-vector: a filtered query widens to four times the probes, not without bound

With no upper bound Lance prepares every partition for the late search
even when the minimum probes already hold k rows, and a filtered query
through the index cost as much as the exact scan. A query that names its
own probe count gets exactly that count.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* test/s3vectors: report recall through an index without asserting it

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test/s3vectors: count a hit tying the k-th best score toward recall

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* admin: lance_vector_index runs in the Lance lane

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* weed-worker: check a table's index before counting its rows

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* test/s3vectors: make build and clean cover the worker leg's binaries

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* test/s3vectors: allow float32 rounding at the recall cutoff

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test/s3vectors: count a repeated hit key once toward recall

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test/s3vectors: credit cutoff ties only up to the slots the top-k leaves

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test/s3vectors: assert every filtered hit passes the filter, indexed or not

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* weed-vector: forget the shared session when a table was dropped and created again

The session keys index lists and transactions by dataset version, and a
table created again at the same location restarts its versions. An open
that finds an older version than the newest seen at that URI, or the same
version committed later, replaces the session and opens again.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* weed-vector: test an open after another engine created the table again

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* weed-vector: replace the session in the same lock that records a recreated table

A second open of the recreated table could see the new version recorded
and the old session still shared, and keep the dropped table's index list.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* weed-worker: skip a vector table whose rows cannot be counted

Detection returned on the first table whose count_rows failed, so one
unreadable table kept every other table from being proposed. It now skips
that table like an open or index failure.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* grafana: panel for vector indexes the Lance worker built

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* seaweed-vector: size default probes from the largest index on the column

A sharded build gives the column one index name per shard. The first
name alone could be a small shard, and its probe count would then be
spent on a larger one.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* seaweed-vector: skip the probe lookup when a query is given no segments

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* weed-vector: open every table in its own Lance session again

One session shared by every open keeps a table's index loaded between
queries, but Lance keys what it caches by location and version, so a
table dropped and created again at the same location could be served
the old table's index list. Telling the two apart from version and
commit time needs a map that only grows, a retry loop and clocks that
agree across engines. A private session per open, as before, has no
such state. Measured at 50k x 384 through an index, eight clients: 164
queries/s with the shared session, 137 without; one client is unchanged.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* weed-vector: reject --nprobes 0

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* seaweed-vector: read the partition count from the opened index

Lance's index statistics took 3.6 ms a query on a 200k-row table with
447 partitions. Opening the index and asking it takes 0.4 ms, and the
scan that follows finds the index already open.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* ci: give the S3 Vectors worker legs the vendored protoc

The worker workspace has no protoc of its own for lance's build
scripts, so the worker legs failed at their first build.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* weed-worker: give a vector index build an hour before the admin times it out

The job set no execution timeout, so the admin's default of 90 seconds
applied, and a build that trains on a large table would be reported as
failed and retried while it was still running.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* weed-worker: lance_vector_index proposes at most the admin's max_results

Every qualifying table went into one message however many there were.
The detector stops at the limit the request carries and sets has_more,
as the Go detectors do.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* weed-worker: a vector index job re-checks the table it was proposed for

A table emptied since detection completes without a build, and one
without a vector column fails instead of reporting it already indexed.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* weed-worker: a vector index run window as long as one build

An attempt is capped at the end of the job type's run window plus the
drain grace, and the default window is 30 minutes, so a build could be
cut off before its own hour was up. The window is an hour too, as the
Go iceberg handler sets both.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

---------

Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
2026-11-04 22:16:31 -08:00
Chris LuandClaude Fable 5.1 b5951c5c7c seaweed-vector: probe policy for indexed queries (ANN phase 1) (#12412)
* seaweed-vector: build and extend the vector index

weed_lance_worker::vector_index builds an IVF_FLAT index in the metric the
table was created with, sizes partitions from the row count, and extends an
index with a new segment or a retrain; the core crate re-exports it and adds
the probes a query spends. Tests cover coverage after build, append, retrain
and compaction.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* weed-vector-cli: index build and index optimize

Logs go to stderr so stdout stays the JSON a caller parses.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* test/s3vectors: PERF_INDEX builds the index before the timed queries

The cluster fixture hands out its catalog URL, the Makefile builds
weed-vector-cli, and the perf suite builds the index after ingest and
reports the build time and segment count.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* seaweed-vector: probe a tenth of the index's partitions, more under a filter

A query that names no probe count gets one from the index's partition
count. Lance searches at least that many partitions, and under a prefilter
keeps going until it has k rows; without one it stops there.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* weed-vector: --nprobes overrides the probe policy

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* seaweed-vector: test that a build takes the table's metric

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* test/s3vectors: make clean removes weed-vector-cli

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* seaweed-vector: measure recall with vectors the table does not hold

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* weed-vector: keep the read cache's doc comment on with_read_cache

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* seaweed-vector: optimize only the indices on the vector column

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* seaweed-vector: resolve the query metric with index::distance_metric

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* seaweed-vector: test the L2 metric under the name CreateIndex stores

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* weed-worker: open a table inside a caller's Lance session

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* weed-vector: keep opened indices between queries in one session

Every open used a fresh Lance session, so the index and metadata caches
died with the request and each query loaded the IVF index again.
--index-cache-mb sizes the shared cache, 512 MiB by default.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* test/s3vectors: PERF_CLUSTERS draws the perf suite's vectors around shared centres

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* seaweed-vector: test that compaction drops the deleted key from tied hits

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* seaweed-vector: test that an append-only optimize serves the new row

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test/s3vectors: record PERF_NOISE in the perf report config

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* seaweed-vector: a filtered query widens to four times the probes, not without bound

With no upper bound Lance prepares every partition for the late search
even when the minimum probes already hold k rows, and a filtered query
through the index cost as much as the exact scan. A query that names its
own probe count gets exactly that count.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* test/s3vectors: report recall through an index without asserting it

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test/s3vectors: count a hit tying the k-th best score toward recall

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test/s3vectors: allow float32 rounding at the recall cutoff

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test/s3vectors: count a repeated hit key once toward recall

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test/s3vectors: credit cutoff ties only up to the slots the top-k leaves

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test/s3vectors: assert every filtered hit passes the filter, indexed or not

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* weed-vector: forget the shared session when a table was dropped and created again

The session keys index lists and transactions by dataset version, and a
table created again at the same location restarts its versions. An open
that finds an older version than the newest seen at that URI, or the same
version committed later, replaces the session and opens again.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* weed-vector: test an open after another engine created the table again

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* weed-vector: replace the session in the same lock that records a recreated table

A second open of the recreated table could see the new version recorded
and the old session still shared, and keep the dropped table's index list.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* seaweed-vector: size default probes from the largest index on the column

A sharded build gives the column one index name per shard. The first
name alone could be a small shard, and its probe count would then be
spent on a larger one.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* seaweed-vector: skip the probe lookup when a query is given no segments

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* weed-vector: open every table in its own Lance session again

One session shared by every open keeps a table's index loaded between
queries, but Lance keys what it caches by location and version, so a
table dropped and created again at the same location could be served
the old table's index list. Telling the two apart from version and
commit time needs a map that only grows, a retry loop and clocks that
agree across engines. A private session per open, as before, has no
such state. Measured at 50k x 384 through an index, eight clients: 164
queries/s with the shared session, 137 without; one client is unchanged.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* weed-vector: reject --nprobes 0

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* seaweed-vector: read the partition count from the opened index

Lance's index statistics took 3.6 ms a query on a 200k-row table with
447 partitions. Opening the index and asking it takes 0.4 ms, and the
scan that follows finds the index already open.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

---------

Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
2026-11-04 21:07:13 -08:00
Chris LuandClaude Fable 5.1 f0f0ba85ae seaweed-vector: build a vector index from the CLI and measure it in the perf suite (ANN phase 0) (#12411)
* seaweed-vector: build and extend the vector index

weed_lance_worker::vector_index builds an IVF_FLAT index in the metric the
table was created with, sizes partitions from the row count, and extends an
index with a new segment or a retrain; the core crate re-exports it and adds
the probes a query spends. Tests cover coverage after build, append, retrain
and compaction.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* weed-vector-cli: index build and index optimize

Logs go to stderr so stdout stays the JSON a caller parses.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* test/s3vectors: PERF_INDEX builds the index before the timed queries

The cluster fixture hands out its catalog URL, the Makefile builds
weed-vector-cli, and the perf suite builds the index after ingest and
reports the build time and segment count.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* seaweed-vector: test that a build takes the table's metric

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* test/s3vectors: make clean removes weed-vector-cli

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* seaweed-vector: optimize only the indices on the vector column

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* seaweed-vector: resolve the query metric with index::distance_metric

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* seaweed-vector: test the L2 metric under the name CreateIndex stores

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test/s3vectors: PERF_CLUSTERS draws the perf suite's vectors around shared centres

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* seaweed-vector: test that compaction drops the deleted key from tied hits

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* seaweed-vector: test that an append-only optimize serves the new row

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test/s3vectors: record PERF_NOISE in the perf report config

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test/s3vectors: report recall through an index without asserting it

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test/s3vectors: count a hit tying the k-th best score toward recall

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test/s3vectors: allow float32 rounding at the recall cutoff

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test/s3vectors: count a repeated hit key once toward recall

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test/s3vectors: credit cutoff ties only up to the slots the top-k leaves

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test/s3vectors: assert every filtered hit passes the filter, indexed or not

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
2026-11-04 19:25:38 -08:00
Chris LuandClaude Opus 5.5 8350f297af seaweed-vector: dev plan for ANN search over S3 Vectors indexes (#12408)
* seaweed-vector: dev plan for ANN search over S3 Vectors indexes

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* seaweed-vector: dev plan: one probe count per query, a budget that can shrink

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* seaweed-vector: dev plan: smaller deltas are searched in full only up to the probe count

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* seaweed-vector: dev plan: drop the private issue reference

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* seaweed-vector: dev plan: phase 1 probes use the Design count

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* seaweed-vector: dev plan: status after phases 0 to 3, and what the measurements changed

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* seaweed-vector: dev plan: name the Enterprise repository, not the private one

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* seaweed-vector: dev plan: the Enterprise repository by that name

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* seaweed-vector: dev plan: a filtered query widens to four times the probes, not without bound

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* seaweed-vector: dev plan: throughput through the index at 50k and 200k

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* seaweed-vector: dev plan: union coverage over shard names, and an explicit merge at the delta limit

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* seaweed-vector: dev plan: status table names branches, not pull requests

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* seaweed-vector: dev plan: maximum_nprobes is always four times the count

Lance reads maximum_nprobes only under a prefilter, and the phase 1
policy sets it the same way for every query.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* seaweed-vector: dev plan: the retrain follows the partition count, not a recorded row count

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* seaweed-vector: dev plan: phase 3 is open like the other phases

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* seaweed-vector: dev plan: the retrain stops past 2,048 partitions

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* seaweed-vector: dev plan: the merge at the delta limit takes the name's delta count

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
2026-11-04 19:25:35 -08:00
github-actions[bot] cd10b4a5c1 docs: regenerate star history chart 2026-11-05 02:07:12 +00:00
Chris Lu 683c5ad4d3 admin: regenerate the plugin pages' templ output from weed/admin (#12405)
Both were generated at the repo root, so their embedded source paths read
weed/admin/view/... where every other admin template, generated by
weed/admin's Makefile, reads view/.... No markup changes.

Claude-Session: https://claude.ai/code/session_011UpvafcfUuC2mLxNAXLQJi
2026-11-04 11:47:36 -08:00
Chris LuandClaude Opus 5.5 7047d3947a admin/plugin: fail a dispatch its restarted worker no longer lists (#12402)
* admin/plugin: fail a dispatch its restarted worker no longer lists

A job dispatched to a worker that then restarts dies with the old
process. The worker re-registers under the same id on a new stream and
its heartbeats stop listing the job, but nothing ended the pending
execution: the dispatcher kept waiting and held the executor slot until
the attempt's execution timeout.

Each pending execution now records the stream it went out on. An
itemized heartbeat from a newer stream of that worker that leaves the
job out twice in a row fails it through the expire path, so the
dispatch returns, the slot is released and the retry machinery takes
over. Jobs on the heartbeat's own stream, jobs the new stream still
lists, and heartbeats that count slots without itemizing them are left
alone.

Claude-Session: https://claude.ai/code/session_011UpvafcfUuC2mLxNAXLQJi

* admin/plugin: let a partly itemized listing clear a dispatch's misses

A heartbeat that counted more slots than it itemized returned before the
listed jobs had their misses reset, so an omission, a partial heartbeat
listing the job, and another omission failed a job the worker was still
running.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* admin/plugin: look up the worker's stream before tracking the attempt

A dispatch to a worker with no stream returned after the job was already
recorded as assigned, and that record kept new proposals with the same
dedupe key out until the stale-job sweep expired it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* admin/plugin: fail the attempt whose request could not be sent

The dispatch now sends on the stream it looked up before tracking the
attempt, so a worker that reconnects in between makes the send fail.
That returned with the job still tracked as assigned, and with the
worker back nothing closed the record for the 24h stale timeout while
its dedupe key kept new proposals out. The attempt is now completed as
failed, after it is unregistered so the completion does not wait on a
channel nobody reads.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* admin/plugin: leave a send failure to the expiry that already claimed it

The send-error path and an orphan expiry could both complete the same
attempt and record two failed runs for it. The send-error path now
completes the attempt only if no expiry has marked it failed, decided
under the same lock that unregisters it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* admin/plugin: record a send failure whose entry Shutdown already removed

Shutdown closes shutdownCh before it unregisters pending dispatches, so a
blocked send returns while its map entry is gone. Read the expiry claim
from the dispatch's own state instead of the map lookup, so the attempt
is still failed and does not stay assigned.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* admin/plugin: match a padded job id against its trimmed listing

Heartbeat work ids are trimmed before the orphan check, so compare the
dispatch's job id trimmed too; otherwise a job id with surrounding
spaces never matches its own listing and the running job is failed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* admin/plugin: skip orphan misses once a completion was handed over

The buffer length only shows a completion the dispatcher has not taken
yet; after it takes one, the attempt stays registered until the dispatch
returns. Mark the hand-off under the pending lock and check that flag
instead, so heartbeats in that gap cannot fail a finished attempt.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-04 11:45:41 -08:00
Chris LuandClaude Opus 5.5 3d411c6a6c volume: report EC file and delete counts in incremental shard messages (#12404)
* seaweed-volume: report EC file and delete counts in heartbeat deltas

The full EC heartbeat already carries file_count and delete_count, but the
incremental new/deleted shard messages built in
collect_ec_shard_delta_messages left both at zero. A volume the master first
learns of through a delta then reads 0 files until the next full EC
heartbeat. Fill both from file_and_delete_count, the same source the full
heartbeat uses, computed once per volume rather than once per shard.

Claude-Session: https://claude.ai/code/session_011UpvafcfUuC2mLxNAXLQJi

* volume: report EC file and delete counts in mount notifications

Go parity with the Rust delta change: the NewEcShardsChan messages sent on
MountEcShards and on startup shard load now carry FileCount and DeleteCount
from EcVolume.FileAndDeleteCount, the same values the full EC heartbeat
reports, instead of leaving them at zero.

Claude-Session: https://claude.ai/code/session_011UpvafcfUuC2mLxNAXLQJi

* seaweed-volume: leave count-only EC changes out of the shard delta

The delta messages now carry file_count and delete_count, so the whole-message
comparison re-sent every shard of a volume after any delete, and the master
broadcast each one as a new EC vid. Compare the shard fields only; the full EC
heartbeat already carries the counts.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* topology: mount messages now carry EC counts, only older servers omit them

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* storage: guard the captured EC mount messages in the disk-type tests

The drain goroutine appended to the slice that waitForShardMsg polled with no
lock, which go test -race reports.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* storage: test EC counts on the startup notification and repeated .ecj ids

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* seaweed-volume: move file_count too in the count-only delta test

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-04 10:43:04 -08:00
Chris LuandClaude Opus 5.5 3cbace2471 s3: expose ReadAcp/WriteAcp in shell and admin UI, log ACL-upload denials (#12396)
* shell: accept ReadAcp and WriteAcp in per-bucket and service account actions

s3.bucket.access, s3.anonymous.set and s3.serviceaccount.create rejected
ReadAcp and WriteAcp, so the ACL actions the S3 gateway enforces could not
be granted through them. Share one canonical action table across the
three commands and list the ACL actions in their help.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* shell: list ReadAcp and WriteAcp in s3.configure and s3.circuitBreaker help

The -actions usage of both commands stopped at Admin, so the ACL actions
the gateway enforces could only be found in the source. s3.circuitBreaker
now prints the actions it actually accepts.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* admin: offer ReadAcp and WriteAcp in the object store user permissions

The create and edit user forms only offered Read, Write, List, Tagging and
Admin among the bucket permissions, so an identity could not be given the
ACL actions from the UI even though uploads with x-amz-acl need WriteAcp.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3api: log the WriteAcp denial of an upload that sets an ACL

An upload or copy that carries x-amz-acl or an x-amz-grant-* header also
needs WriteAcp. When that check denies, the client only sees a bare
AccessDenied and nothing reached the log at the default verbosity, so
the header was never suspected. Log the identity and the missing action.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3api: quote the key in the ACL upload denial log

An object key can carry a decoded newline, which let a caller add forged
lines to the default log. Quote the bucket and key, and word the message
so it also fits a policy deny, not only a missing grant.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* shell: say ACL uploads need both Write and WriteAcp

The s3.configure example granted WriteAcp alone, which still fails the
upload's Write check.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* shell: parse S3 action lists with one shared helper

s3.bucket.access, s3.anonymous.set and s3.serviceaccount.create each had
their own validation loop, so a test of the table did not cover the
parsing the commands run. They now call canonicalActionList, which skips
empty entries and repeats. s3.anonymous.set validates before contacting
IAM, and s3.bucket.access still rejects a list with no actions.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* admin: keep bucket-scoped actions the user form cannot show

When a user has both global and bucket-scoped actions, the edit form
shows only the global ones, and saving replaced the identity's actions
with what the form rebuilt, deleting the scoped grants. Carry those
actions through the save.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* admin: list hidden bucket grants in the user edit form

Restoring the bucket-scoped grants the form cannot show kept them active
even when an admin narrowed the user's permissions, with no way to see
or drop them. Show them as checked boxes under Other Bucket Permissions;
unchecking one removes it on save.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* shell: reject an s3.anonymous.set -access list with no actions

A blank list such as ",," parsed to nothing and cleared the bucket's
anonymous grants. Use "none" for that, as s3.bucket.access does.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3api: skip the ACL upload denial log for anonymous requests

Anonymous uploads with ACL headers can be sent by anyone, so logging
each denial at the default level lets unauthenticated clients flood the
log. Keep the line for named identities.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* admin: count kept bucket grants when checking an edited user for permissions

The empty check ran before the checked Other Bucket Permissions were
merged, so a user left with only those grants could not be saved.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* admin: list bucket grants of Admin users in the user edit form

parseBucketPermissions returned before reading scoped actions when the
user had Admin, so removing Admin and saving dropped grants such as
WriteAcp:bucket.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* admin: keep unpaired bucket grants as separate checkboxes in the user edit form

The form saves every selected permission on every selected bucket, so
grants such as ReadAcp:a with WriteAcp:b came back as all four pairs.
Show the grants through the picker only when they cover that full
product; otherwise list each grant on its own. Opening the form also
resets the bucket scope left over from the previous user.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* admin: keep grants the edit pickers have no option for

A permission x bucket product was shown through the pickers even when a
bucket was not in the bucket list (deleted, prefix or wildcard grants)
or a permission had no picker option (DeleteBucket, PutBucketQuota).
Saving then dropped those grants. Such grants now go to the checkbox
list, global ones included, and are submitted unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* shell: name the s3.bucket.access anonymous example by what it grants

Read,Write,List leaves out ReadAcp and WriteAcp, so it is not full
access.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* admin: keep unlisted global grants of Admin users in the user edit form

parseBucketPermissions returned for Admin before splitting global grants
into picker options and the rest, and only raw option values were
re-selected. Saving an Admin user dropped s3tables:* and grants with no
picker option such as DeleteBucket. Admin users now use the same split.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* admin: keep the /* suffix of bucket grants in the user edit form

parseBucketPermissions dropped a trailing /* before matching bucket
options, so saving rewrote ReadAcp:bucket/* as ReadAcp:bucket. The bare
form also matches bucket-level requests such as GET ?acl, so the save
widened the grant. A /* bucket is not a picker option, so the grant now
stays in the checkbox list as written.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* shell: test that s3.bucket.access saves ReadAcp and WriteAcp grants

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-04 09:55:54 -08:00
Chris LuandClaude Opus 5.5 c437791991 s3: reject POST Object header fields with control characters (#12400)
A Content-Type form field such as "\n" was copied verbatim into the
multipart part header sent to the volume server, which broke the
framing and surfaced as a Content-MD5 mismatch and a 500. Form fields
promoted to request headers must be valid header values, as on a PUT;
otherwise answer 400 MalformedPOSTRequest.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-04 09:43:48 -08:00
0742d316aa fix(s3): accept fresh admin tokens when context carries stale authorization (#12390) (#12397)
* fix(s3): accept fresh admin tokens when context carries stale authorization (#12390)

Signed-off-by: Mohd Quamar Tyagi <104281681+Tyagiquamar@users.noreply.github.com>

* security: share the admin Bearer check between S3 and admin-plane peers

Move the authorization scan into security.CheckGrpcAdminToken, built on
IsFilerAdminToken, and call it from CheckGrpcAdminPeer and the S3 IAM cache
check. Both now accept a valid admin token in any authorization value
instead of S3 re-implementing the decode and claim checks.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* filer: accept a fresh admin token after a stale one on IAM gRPC

The S3 IAM cache check is meant to mirror the filer's, so route the filer's
checkAdminAuth through security.CheckGrpcAdminToken too. IamGrpcStore appends
its token to the caller's context, so a forwarded older token would otherwise
still shadow it here.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* security: parse at most four authorization values per admin call

Scanning every value let one request repeat a junk Bearer token (cheap over
HPACK) tens of thousands of times and burn about half a second of JWT parsing
on an unauthenticated port. Real callers send their own token plus at most one
forwarded from their context.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* credential: match the other WithS3InternalAdminAuth replace fix

The plugin worker fix makes the same set-instead-of-append change to
WithS3InternalAdminAuth with different code and a different test at the same
spot, so whichever lands second would conflict. Use the identical code and
test so the two merge cleanly in either order.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* security: test an expired admin token ahead of a fresh one

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* security: test that a same-key session token is not an admin token

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* filer: test IAM gRPC with only invalid authorization values

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Signed-off-by: Mohd Quamar Tyagi <104281681+Tyagiquamar@users.noreply.github.com>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-04 09:42:25 -08:00
Chris LuandClaude Opus 5.5 51ee165279 ec: rebuild a needle when other shards are corrupt over its range (#12401)
* storage: search corrupt shard sets when an EC needle fails to rebuild

A needle whose reassembled bytes fail to parse is rebuilt with its own
shards excluded. When other shards are corrupt over the same range, that
decode reads them as inputs, and the retry dropped only one input at a
time, so two or more extra corrupt shards left the needle unreadable even
within parity.

After the single pass fails, try every set of up to ParityShards shards
that holds a suspect as the corrupt set, smallest first. Suspect ranges
inside the set are rebuilt from the shards outside it and the rest keep
their bytes, so a clean suspect can still feed the decode. Sets whose
decodes read the same inputs are tried once, and the search is bounded
by set count and by rebuilt bytes past the single-shard sets.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* seaweed-volume: search corrupt shard sets when an EC needle fails to rebuild

Mirror the Go read path: after the single pass over the suspects fails,
gather each interval from every other shard once and try each set of up
to parity_shards shards holding a suspect as the corrupt set, smallest
first, with the same dedup and bounds. The search runs on the blocking
pool, holding the recovery permit, since it can take many decodes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* ec: charge only multi-shard sets against the rebuild byte budget

The budget check already let single-shard sets through, but each one
was still charged, so a needle spanning many shards could spend the
budget before the first multi-shard set.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* ec: bound the corrupt-set gather to the recovery budget

The search gathered every other shard per interval, so a large needle
spanning several shards held up to Total-1 copies of itself under a
permit capped at the budget. Gather only as many as the budget covers,
but never fewer than the suspects-excluded gather the fallback always
held, so single-suspect recovery keeps every other shard.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* ec: gather at least one decode's worth for the corrupt-set search

A needle over more shards than parity skips the single pass, and its
suspects-excluded width can fall below DataShards, so the bounded
gather left even one corrupt shard unrecoverable. Floor the width at
DataShards, which the per-suspect rebuild this search replaced read.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* ec: rebuild one suspect at a time when the search gather is narrow

Flooring the search gather at DataShards made a large needle over more
shards than parity hold ten copies of itself, where the per-suspect
rebuild it replaced held one interval's fan-out. Keep that rebuild
after the search, and skip the search when the budget leaves fewer
inputs than one decode. Parity slack still widens the gather to every
shard outside the suspects, as the dropped-input retry did.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* ec: keep the dropped-input retry and run the set search within budget

A narrowed search gather skipped only each interval's own shard, so it
could read corrupt suspects in place of healthy shards the old retry
kept, and its byte bound could stop before that retry's hypotheses.
Restore the dropped-input retry as it was, move the search into its
own function after it, and run the search only when every interval's
full gather fits the recovery budget. Past the budget a read takes the
same paths, with the same footprint, as before the search.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-04 09:41:21 -08:00
Chris LuandClaude Opus 5.5 8f29e38837 grafana: chart EC reconstruction and other unplotted metrics (#12406)
* grafana: chart EC intervals served by reconstruction

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* grafana: chart volume server storage IO errors and quarantine

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* grafana: chart frozen volume growth and duplicate volume ids

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* grafana: chart filer metadata log gaps and replay failures

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* grafana: add failed bytes to filer sync throughput

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* grafana: chart remote mount cache reads and evictions

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* grafana: chart plugin worker detection duration

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* grafana: show duplicate volume ids whose first detection was the only one

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* grafana: keep cluster apart in the master growth and duplicate id panels

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-04 09:37:43 -08:00
d6c65cd924 volume: count EC intervals served by reconstruction (#12392)
* volume: count EC reads that fall back to reconstruction

When a needle read can't read an EC shard interval directly (the shard isn't
mounted here and no holder answers), both volume servers rebuild it from the
other shards and the read still succeeds, with nothing to show for it.

Count those fallbacks in SeaweedFS_volumeServer_ec_degraded_read_total{result}
(success, failure), one per interval, at the one place each server falls back:
Go readOneEcShardInterval, Rust fetch_one_interval. Corruption repair and scrub
aren't counted. A line binding each one to the needle and the volume's encode
generation is logged at V(3) in Go and debug in Rust.

* volume: name the EC counter for what it counts

Reed-Solomon reconstruction can't tell corrupt inputs from good ones, and the
needle is validated later, so "success" only ever meant the reconstruction
produced data. Rename the counter to
SeaweedFS_volumeServer_ec_reconstructed_interval_total and the log line to
"ec reconstructed interval", say so in the help text, and pin the behaviour
with a test: an interval rebuilt from corrupt parity counts success while the
read itself fails.

* storage: restore stderr when the log-line test stops early

A fixture failure in readEcNeedle ends the test through t.Fatalf, which
skipped restoring os.Stderr and closing the pipe, so later tests wrote
into a pipe nobody read.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-04 08:30:41 -08:00
15ff5c113a fix(s3tables): validate policy conditions and fail safe on unusable ones (#12389)
* s3tables: validate policy conditions and fail safe on unusable ones

* s3vectors: reject a condition operator with no keys

The shared evaluator now treats a keyless operator as unusable, so a
stored keyless Allow would never grant while the write succeeded.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3vectors: reject an Allow whose Condition is null

A null Condition decodes as no Condition, so the Allow granted
unconditionally. A null Condition on a Deny only widens the veto and
stays accepted, as in the S3 Tables validator.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3tables: treat a NaN numeric condition value as unreadable

strconv.ParseFloat accepts "NaN", but no comparison with it holds, so a
Deny on NumericLessThan never applied and an Allow on NumericNotEquals
matched every request.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3tables: reject condition keys the evaluator never supplies

A condition on a key such as aws:SourceIp reads no request value, so a
Deny on it never applied and an Allow never granted. Accept only the
keys getConditionContextValues resolves, as the vector validator does.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3vectors: require Statement and Effect spelled as written

The evaluator decodes field names regardless of case, so an "effect" or
"statement" key slipped past the null Condition check and the other
raw-key checks while the stored Allow still granted.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3tables: treat an infinite numeric condition value as unreadable

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3tables: a stored Allow with a null Condition never grants

The typed decode reads "Condition": null as no Condition, so an Allow
stored before validation refused it still granted unconditionally, for
table and vector policies alike. Record the literal null while decoding
a Statement and skip such an Allow during evaluation.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3tables: validate a null Condition from the decoded Statement

validatePolicyDocument re-parsed the document to find a literal null
Condition; the Statement decode now records it, so the validator reads
the same flag the evaluator does and nullConditionStatements goes away.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3tables: keep Condition objects merged across key spellings

Decoding Condition through a json.RawMessage kept only the last of
"Condition" and "condition", so an empty object under one spelling
erased the other's restriction. Decode the statement as before and
probe for the literal null separately.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3tables: refuse a Condition split across key spellings

Condition objects under differently cased keys decode into one map, and an
operator they share keeps only the last spelling's keys, so a restriction
under the other spelling was dropped. Validation now refuses a statement
whose Condition is under more than one key, and one stored before that
never grants on an Allow and vetoes on a Deny.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3tables: refuse a Condition key repeated under the same spelling

PolicyDocument re-marshaled each statement through interface{}, and
Statement.UnmarshalJSON counted keys from a map, so both kept only the
last of a repeated Condition key. "Condition":null,"Condition":{} or two
Condition objects sharing an operator passed validation with a restriction
dropped. Statements now decode from their original bytes and the keys are
counted while walking the object.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3vectors: refuse a repeated Condition key in a vector bucket policy

The validator read Condition from a map, which keeps only the last of a
repeated key, so it accepted a statement the evaluator never grants on.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3tables: keep the re-marshaled statement decode, count Condition keys apart

Decoding statements from their original bytes let document order pick
between spellings of every field, where the re-marshal sorted the keys
first, so a stored "effect":"Deny","Effect":"Allow" became an Allow.
Statements decode as before again; only the Condition keys are counted
from the original bytes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3tables: test that validation refuses a split Condition on a Deny

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3vectors: test that validation refuses a repeated Condition on a Deny

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-04 08:13:05 -08:00
Chris LuandClaude Opus 5.5 d727c6970f plugin worker: stop forwarding the stream's admin token to job RPCs (#12391)
* plugin worker: keep the stream's admin token out of job contexts

The bearer token minted when the worker opens its control stream was
attached to connCtx, which every detection and execution context derives
from. Job RPCs therefore forwarded that connect-time token, and when a
job added its own fresh token (s3_lifecycle's LifecycleDelete) the S3
server validated the stale one first and rejected the call once it
expired.

Attach the token only to the WorkerStream call, as the Rust worker does.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* credential: replace an inherited admin token instead of appending one

WithS3InternalAdminAuth appended its fresh Bearer token, so a context
that already carried an authorization value sent both, and the S3
gateway's checkAdminAuth validates only the first. Set the header on a
copy of the outgoing metadata so the fresh token is the only one sent.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* plugin worker: read the captured job metadata after the stream ends

Both result channels could be ready at once, letting select fail a
successful run. The handler reports before JobCompleted reaches admin,
so wait for the stream to finish and then read the captured metadata.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* plugin worker: test that dropping the connection cancels running jobs

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-04 05:26:12 -08:00
Chris LuandClaude Opus 5.5 a0f30afe5f s3: keep service account keys when a pushed identity replaces its parent (#12394)
* s3: keep service account keys when a pushed identity replaces its parent

Service account changes push the parent user, whose proto carries only
the user's own keys. The merge replaced the cached parent with that
proto and dropped every service account key until a restart. Read the
parent's service accounts from the credential store so the push
rebuilds them, deleted and disabled ones included.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3: test that a parent identity push keeps its service account keys

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3: test a sibling service account survives a deletion push

Also check the restored secret and status, not just the key.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-04 05:26:07 -08:00
Chris LuandClaude Opus 5.5 92bdfeaf29 seaweed-vector: record seaweed-worker-core's toml dependency in Cargo.lock (#12388)
seaweed-worker-core now depends on toml, but the seaweed-vector
workspace lockfile was not refreshed, so cargo build --locked fails
before the S3 Vectors integration tests start.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-03 20:48:56 -08:00
Chris LuandClaude Opus 5.5 ba3fd257a3 filer: clear a dead test replica's reachability record before closing it (#12387)
* filer: clear a dead test replica's reachability record before closing it

The read path tries hosts that recently failed last. A dead test server
can land on a port an earlier test left marked as failed, so the live
replica answered first, nothing failed and the location refresh under
test never ran. Have the dead replica answer once through util_http so
its record is cleared before it closes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* filer: close the dead test replica when reaching it fails

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* filer: clear the live test replica's reachability record too

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-03 20:47:43 -08:00
329ccb8bc0 master: tell a warming-up Assign or LookupVolume client when warm-up ends (#12386)
* master: tell a warming-up Assign client when warm-up ends

During warm-up the master refuses Assign with Unavailable and now attaches
the remaining warm-up as a gRPC RetryInfo detail. RetryWithBackoff may
shorten a backoff step to that hint plus 50 ms, never lengthen it, and only
for a valid, positive hint, so a writer retries right after warm-up ends
instead of up to one backoff step later. The warm-up itself is unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* master: jitter the retry-hint slack and hint LookupVolume too

Per review: the slack added to a server hint is now random in
[50ms, 300ms], so clients refused at the same moment do not all retry
in one instant. The warm-up refusal of LookupVolume carries the same
RetryInfo hint as Assign; the masterclient's lookup retry already goes
through RetryWithBackoff, so it lands at warm-up end too.

Tests: the slow guard tests are replaced by one sleep-free table test
of the step logic (zero, negative, huge, malformed, valid and wrapped
hints, a hint longer than the step, none). One timing test of the
shortened step stays, plus a jitter-bounds test and a LookupVolume hint
test. The no-hint control test, which was timing-sensitive, is dropped.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* master: name the shared warm-up test and file the lookup test by topic

Rename the no-remaining test to TestWarmingUpErrorWithoutRemainingHasNoHint,
since the helper now serves Assign and LookupVolume, and move the
LookupVolume hint test next to the other LookupVolume warm-up tests.
Tests only.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* util: read the retry hint through ServerStatus

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* master: bound the warm-up hint in tests by the remaining warm-up around the RPC

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test: leave more scheduler margin in the retry-hint timing tests

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* util, master: trim retry-hint comments

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test: widen the retry-hint timing bounds to just under the unhinted wait

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test: require a refused warm-up call before the hinted assign retry

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test: check the hinted retry wait through a test timer, not the wall clock

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test: move the assign warm-up end to 2.6s for more hinted-retry headroom

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test: require the hinted retry to wait out its timer

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
2026-11-03 20:46:58 -08:00
Chris LuandClaude Opus 5.5 cd82e663ad admin: credit an unitemized slot to a pending job only after a later heartbeat (#12385)
* admin: credit an unitemized slot to a pending job only after a later heartbeat

A pending reservation the worker does not list took one of the slots
the worker counts without itemizing, even when that heartbeat came
before the reservation and could not have counted the job. The slot
belonged to unknown work, so the admin undercounted it and could admit
one job too many, past a job type's per-worker limit, until the next
heartbeat.

A reservation now records when it was made, and claims an unitemized
slot only from a heartbeat received at or after that time.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* admin: time unitemized-slot credit by the heartbeat, not the last hello

A reconnect's hello moves LastSeenAt but keeps the old heartbeat, so a
pending reservation could claim a slot counted before it existed, and a
completed job's echo could end without a new heartbeat. Record when the
heartbeat itself arrived and compare reservations against that.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* admin: a completed job claims no slot from a heartbeat before its reservation

A job that completes before any heartbeat counts it has no echo, so the
slot the last heartbeat counted is other work. Crediting it to the job's
type let another job type pass its per-worker limit until the next
heartbeat.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-03 20:45:39 -08:00
github-actions[bot] 995084f71e docs: regenerate star history chart 2026-11-04 00:54:17 +00:00
Chris LuandClaude Opus 5.5 5dfda3645e admin: count executor slots per job type and stop lanes waiting on capacity they cannot free (#12384)
* admin: count executor slots per job type, deduped against the heartbeat

per_worker_execution_concurrency is a per-job-type setting, but the
scheduler compared it against the worker's execution_slots_used, which
counts every job type. One s3_lifecycle job a worker reported running
after a reconnect made admin_script (per-worker 1) undispatchable on a
worker with three free slots. A reservation and the heartbeat entry for
the same job were also charged as two slots, so a per-worker limit of N
let N-1 jobs run.

Reservations now carry the job id and job type. The per-type limit is
checked against that type's work alone, the worker's slot total against
work of every type, and a heartbeat entry covered by a reservation is
counted once. Per-lane reservation pools go away: the per-type count is
what kept lanes from starving each other.

* admin: stop a lane waiting on executor capacity it cannot free

A queued job no executor could take was retried every backoff until
job_type_max_runtime ran out, up to 30 minutes, while the default lane
sat in executing and every other job type in it idled.

The dispatch loop now asks what holds the capacity. Slots held by the
lane's own jobs release when they complete, so it keeps waiting. Slots
held by other lanes or by work this admin never dispatched clear on a
schedule the lane has no view of, so the queued jobs are canceled at
once with a skipped_no_executor activity naming the worker, its slot
count and the job types holding it, and the next detection proposes
them again. A missing worker is waited for only the orphan grace.

* admin: cancel a job on its worker when the admin expires it

Expiring a job, by the stale sweep or by the operator's expire endpoint,
only failed the tracked record. The worker never heard about it and kept
running the job with its slot taken, so a job stuck in "starting" held a
slot for over a day and, with the old slot accounting, blocked a whole
lane.

Expiring a job now also sends the worker a cancel for it, so the slot
comes back on the next heartbeat.

* admin: count a completed job's heartbeat echo as its lane's own slot

The worker sends JobCompleted before the heartbeat that frees the slot, so
right after a dispatch goroutine releases its reservation the job still
shows in the worker's running work. That entry was counted as foreign, and
a lane with per-worker 1 gave up and canceled the rest of its queue after
the first job. A released reservation now stays for completedJobEchoGrace
and claims that heartbeat entry as the lane's own load.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* admin: let the admin's own jobs claim slots a worker does not itemize

The Rust worker reports only a slot count, every 10s, and the Go worker
counts a slot one heartbeat before it lists the job. Those slots were
charged on top of the reservations behind them, and once a job completed
its stale count read as foreign work, so the lane gave up its queue.
Unitemized slots now go first to this admin's unlisted reservations and
recent completions; the rest are of unknown type and count against every
type's per-worker limit, not only the slot total.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* admin: give up pinned jobs whose busy home a free worker cannot relieve

A job homed on a replica-host worker stays on that home while its queue is
shorter than LocalQueuePin, so a free remote worker cannot take it. The
outlook treated that free worker as a reason to wait, and a home held by
foreign work parked the lane for its whole window. A free worker now only
holds the wait for the orphan grace, after which the held homes decide.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* admin: let a redispatched job's reservation replace its released one

A job the worker turns away for capacity is released, kept as an echo, and
queued again under the same id. Once the new attempt ran, both the echo and
the new reservation matched its heartbeat entry and it counted as two
slots. Reserving a job now drops what an earlier reservation of it left.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* admin: credit a count-only slot to a completed job only until the next heartbeat

A released reservation claimed an unitemized slot for its full 30s grace,
so on a count-only heartbeat other work of another type was charged to the
completed job's type. A heartbeat that arrives after the release already
reflects the freed slot; from then on the slot is of unknown type.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* admin: name unitemized slots in the no-capacity reason

holders() listed only the job types the worker itemized, so a worker with
named and count-only work read as "2/2 slots used by <type>".

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* admin: keep the blocked clock running until a reservation succeeds

A popped job whose reservation then failed reset the clock before the
failure pushed it back, so pop-then-fail cycles never aged the block.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* admin: resend an expiry cancel the worker did not receive

Expiring a job sends its worker one cancel. When the worker is
disconnected, or its outgoing buffer stays full past the send timeout,
that cancel is dropped. The record is already failed, so later sweeps
and expire calls skip it, and a worker that kept the job running (a
detached job across a reconnect, or a jammed stream) holds the slot
until the job ends on its own.

The admin now holds an undelivered expiry cancel and sends it again on
the worker's next heartbeat that still lists the job. It is forgotten
when that heartbeat no longer lists the job, and skipped when the job is
active again, since a new attempt reuses the job id.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* admin: cancel an expired attempt by its request id

The expiry cancel named the job id, and a retry reuses it. The worker
keys its cancel funcs by both ids, so a cancel arriving after a retry
started on the same worker stopped the retry. The held resend also gave
up on any job that was active again, and the expired attempt's own late
progress marks the job running, so a worker that buffered progress
while disconnected kept the expired job.

The cancel now names the attempt's request id, as the other execution
cancels do, and the held resend no longer depends on the tracked state.
Only a cancel that can name nothing but the job id still skips a job
that is active again.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* admin: ignore late reports from an attempt the admin already ended

Expiring a job now cancels it on its worker, and the worker answers
with its own JobCompleted for that attempt. That overwrote the expiry's
failure with the cancel error and logged a second run record under the
same run id. Late progress from the expired attempt likewise reopened
the record as running.

A progress update or completion whose request id names an attempt the
record already closed is now dropped after the dispatcher's waiter is
told. A retry runs under a new request id, so its reports still apply.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-03 15:10:26 -08:00
Chris LuandClaude Opus 5.5 ca15fdbff6 remote.uncache, remote.meta.sync, remote.cache: keep local writes that land while they run (#12382)
* remote.uncache: skip a file overwritten since the walk read it

The uncache cleared the chunks of the entry its directory walk returned and
wrote it back unconditionally. A file overwritten after the walk read it
was reset to the remote pointer, and its new chunks were dropped, so an
acknowledged write was lost. Write back only while the filer still holds
the entry as read, and report the file as skipped otherwise.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* remote.meta.sync: keep files written locally after the directory listing

meta.sync lists each local directory once and works from that listing.
A changed remote file was written back over the listed entry
unconditionally, and a name missing from the listing was created without
O_EXCL, so a file written locally in between was replaced by the remote
pointer either way. Update only while the filer still holds the listed
entry, create exclusively, and report such files as skipped. The create
now also surfaces errors the filer returns in its response.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* remote.cache: keep files written locally after the lookup

The metadata step looks each file up and then either rewrites it from the
remote unconditionally or creates it without O_EXCL, so a local write
landing between the lookup and the write was replaced by the remote
pointer. Reuse doSaveRemoteEntry, which updates only while the filer still
holds the looked-up entry, create exclusively, and report such files as
skipped.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* remote.meta.sync: do not descend into a name a local file took

When the exclusive create of a remote directory finds the name taken, the
sync still descended into it, and creating the remote children under a
local file would turn that file into a directory. Descend only when the
name now holds a directory.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* remote.meta.sync: delete only the file version it listed

The type-change and -deleteStale paths removed a remote-backed file
unconditionally, so a local write landing after the directory listing was
deleted with it. Condition the delete on the listed mtime, as fs.verify
and volume.fsck do; the filer then keeps a file written since.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* remote.meta.sync: condition directory and mtime-less deletes too

The listed-mtime cutoff was only sent for files, so a remote-backed
directory replaced locally by a file between the emptiness check and the
delete was removed unconditionally, and so was a file listed without an
mtime. Send the cutoff for directories as well, and for an entry listed
without an mtime keep it once a write has set one.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* filer: test that DeleteEntry applies the request's mtime cutoff

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* remote.meta.sync: keep the listed nanoseconds as the cutoff of an epoch mtime

An entry listed at the epoch with a sub-second mtime was sent a 1 ns
cutoff, so the filer kept it unchanged although the old code deleted it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* remote.meta.sync: report an entry the filer kept instead of deleting it

The filer answers a delete it skipped under the mtime cutoff with
success, so the sync printed (delete) for an entry a local write kept.
deleteLocalEntry now looks the name up after the delete and prints the
outcome, and a replaced file that was kept is not recreated.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-03 14:53:42 -08:00
Chris LuandClaude Opus 5.5 b0c6297041 s3api: per-bucket Block Public Access (BlockPublicAcls, IgnorePublicAcls) (#12381)
* s3api: store and serve a bucket's PublicAccessBlock configuration

PutPublicAccessBlock, GetPublicAccessBlock and DeletePublicAccessBlock
answered NotImplemented. Store the configuration in the bucket entry's
extended attributes next to the ownership controls, carry it in the
cached BucketConfig, and answer Get on an unset bucket with
NoSuchPublicAccessBlockConfiguration like S3 does.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3api: refuse public ACLs when a bucket blocks them

With BlockPublicAcls set, a PutObject, CopyObject, CreateMultipartUpload,
POST Object, PutObjectAcl or PutBucketAcl whose canned ACL or grants name
the AllUsers or AuthenticatedUsers group is refused with AccessDenied.
Private and account grants keep working, so clients that send
x-amz-acl: private on every upload are unaffected.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3api: ignore public ACL grants for anonymous reads under IgnorePublicAcls

A bucket with IgnorePublicAcls no longer lets an AllUsers grant on the
bucket or on an object answer an anonymous read or listing. Bucket
policies and identity permissions are unaffected.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3api: reject PublicAccessBlock bodies with unknown elements

A misspelled element such as BlockPublicAcl was ignored, so the request
succeeded and stored the flag as false. Answer MalformedXML instead, and
store the configuration without the request's namespace.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3api: fail closed when a bucket's public access block cannot be read

A stored PublicAccessBlock that does not parse now blocks and ignores
public ACLs instead of acting as if none were set. PutBucketAcl and POST
Object return the bucket lookup error rather than skipping the
BlockPublicAcls check when the bucket config cannot be loaded.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3api: test BlockPublicAcls on copy, multipart initiate and a PUT-then-GET

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3api: keep stored public access block bytes on unrelated bucket updates

An unreadable stored configuration was replaced with the in-memory
all-true fallback whenever another bucket setting was updated, losing
the original value. Write the key only when the update changes the
public access block.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-03 14:52:20 -08:00
ca4ce2c581 volume: a read plan waits for a vacuum commit instead of answering 404 (#12375)
* volume: a read plan waits for a vacuum commit instead of answering 404

A vacuum commit closes the needle map, swaps the compacted files in and
reloads the map. It holds the data-file write lease for that and takes no
store lock. needle_read_plan, which the HTTP GET path uses, read the needle
map without the data-file read lease, so a GET planned in that window found
no map and answered 404 for a live needle.

needle_read_plan now takes the read lease first, as read_needle_with_option
does, so the plan waits for the commit and then reads the reloaded map.
Go's readNeedle does the same under dataFileAccessLock.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* volume: plan a read with the store guard released

A plan that waits on a vacuum commit's data-file lease under the store
read guard parks store writers (allocate, mount, EC ops) on that commit,
and every store reader queued behind them. Look the volume up under the
guard and plan after releasing it, as Go's readNeedle does, and look the
volume up again if its handle was closed while the plan waited.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* volume: retry a read plan while its handle keeps closing

A plan that waited out an unmount looked the volume up only once more,
so a replacement handle that also closed before its plan took the lease
answered NotFound for a needle a later mount holds. Repeat the lookup
while the planned handle is closed, at most three times, so a closed
handle that stays in the map still ends the loop.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* volume: start the swap-window check once the plan holds the volume

The swap-window test gave the plan 300 ms to answer early, so a reader
thread that started late passed the check without waiting on the
commit. Share the writer test's wait for the plan's volume handle and
start both timed checks after it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* volume: run the swap-window read-plan test on a redb map too

A commit reloads the index through load_index_redb for the redb kinds,
a path the in-memory run never took during the swap window.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* volume: say a needle read plan is resolved under the data-file lease

The plan no longer resolves under the store guard, and resolving it
takes the data-file read lease.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* volume: read the needle through the plan in the read-plan tests

The tests checked only that the plan resolved and then read the needle
through Store::read_volume_needle, so a plan with a wrong offset or size
after the commit still passed. Read through the plan, as GET does, and
compare the data.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* volume: test the read plan's lookup after an unmount closed the handle

No test reached the closed-handle retry, so dropping it left every test
green. Take the lookup as a closure so a test can hand the plan the
handle an unmount closed and then the remounted volume, and check that
the plan reads the needle from the remount.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
2026-11-03 14:50:30 -08:00
Chris LuandClaude Opus 5.5 9b5f7bf65b weed-worker: find jwt.filer_signing.key where the Go worker does, and let the chart deliver it (#12373)
* seaweed-common: share the security.toml search path with the volume server

* weed-worker: read jwt.filer_signing from security.toml and WEED_JWT_* like the Go worker

* helm: give the worker-lance container security.toml and the extra environment

* weed-worker: treat an empty signing key in the environment as unset

Viper ignores an empty environment value, so the Go worker falls back to
the security.toml key when WEED_JWT_FILER_SIGNING_KEY is blank. The Rust
worker took the blank value and connected with no token. An empty
--filer-signing-key / WEED_FILER_SIGNING_KEY also falls through to the
security.toml lookup now.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* weed-worker: read a quoted expires_after_seconds like viper

The Go worker reads jwt.filer_signing.expires_after_seconds with GetInt,
which also accepts a quoted number. The Rust worker dropped a quoted value
and minted tokens with no expiry.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* weed-worker: warn when security.toml cannot be read or parsed

Viper logs a warning when an optional config file is broken. The Rust
worker skipped it silently, so a typo looked like an install with no
signing key. Log the path and the parse message, without the quoted line,
which can hold a key.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* helm: keep POD_NAME and LANCE_NAMESPACE_URL out of worker-lance's extra environment

A duplicate name renders after the chart's entry and wins, silently
changing the Lance worker's --id or --namespace.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* weed-worker: list /usr/local/etc/seaweedfs in the security.toml search path

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* weed-worker: an explicit --filer-signing-key-expires-after-seconds wins

The flag was silently ignored whenever the key was discovered, from
WEED_JWT_FILER_SIGNING_KEY as well as from security.toml. Use it when
given, else the expiry configured beside the discovered key, else zero.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* weed-worker: test where filer_signing looks for security.toml

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* weed-worker: read a float expires_after_seconds as viper's GetInt does

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* weed-worker: cap the minted exp so a huge expiry cannot overflow

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-03 14:49:39 -08:00
Chris LuandClaude Fable 5.1 de4a71665e test/s3vectors: wait for the S3 port before handing out the cluster (#12383)
The fixture waited for the Lance catalog port and the engine, but weed s3
opens its HTTP port last, about half a second after the catalog here. The
open cluster's engine now registers within that window, so
test_engine_without_credentials met a connection refused on the go and rust
single legs. Wait for the S3 port like the other listeners.


Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
2026-11-03 14:08:42 -08:00
Chris Lu ab6aa9b645 admin: let a buffered detection completion win over the stream closing (#12379)
The detection waiter selects on the request's completion channel and on
the stream's done channel. A worker that finishes a scan and then drops
its stream (a restart right after a scan is the usual shape) buffers the
completion into the request before the receive loop sees the EOF, and
the stream's cleanup then closes done. A waiter that reaches its select
after both are ready can take the disconnect branch, and a completed
detection, proposals included, is failed as "detection worker
disconnected" and retried or recorded as an error.

When the stream closes or the liveness tick finds the worker gone, the
waiter now takes a completion that is already queued and returns it as
the success it is; the detector lease still goes, since the worker that
answered is unreachable. A ctx that ended keeps its meaning: the caller
stopped wanting the answer. The wait loop moves into
awaitDetectionOutcome so a test can hand it both channels already
ready; before this change that test fails on about half its runs. A
completion that lands after the waiter gave up stays unmatched, since
the request's pending entry is gone and the next request has a new id.
2026-11-03 12:53:28 -08:00
Chris Lu 836c60b2c9 topology: refresh EC file and delete counts from heartbeats (#12378)
* topology: refresh EC file and delete counts from heartbeats

UpdateEcShards only re-stored a heartbeated EC record when shard
membership changed, so FileCount and DeleteCount stayed at whatever the
first heartbeat after registration carried, while the volume server
reports fresh tombstone counts on every full EC heartbeat. Per-collection
object counts and EC vacuum candidate detection read the stored values.

Compare the counts per (volume, disk) against the record already in
hand and copy them onto the stored record in place, so the refresh
raises no new/deleted shard events and rebuilds no maps. A message
reporting 0/0 did not count (older volume servers, and the incremental
mount message omits the fields), so it keeps the stored values, also
across a membership-triggered rewrite. The incremental path's existing
branch picks counts up the same way.

* topology: swap EC shard records instead of mutating them in place

GetEcShards hands out the stored *EcVolumeInfo pointers and releases
ecShardsLock; CollectionStatistics, CollectionEcVolumeStats, ToDiskInfo
and the master's shard broadcast read fields through them afterwards.
The heartbeat count refresh wrote FileCount/DeleteCount into those
records under the lock, which the readers never take, so it was a data
race and a reader could pair a FileCount from one heartbeat with a
DeleteCount from the next. Disk.AddOrUpdateEcShard had the same write
on the incremental path, and its shard-bits/ratio merge was already
in place.

Stored records are now never written after they are published: a count
refresh, an incremental merge and a shard delete each build a copy
beside the stored record and swap the map entry under ecShardsLock, so
a reader holding the old pointer keeps one heartbeat's snapshot. The
refresh compares against whatever record is current under the lock,
since the incremental path may have replaced it since UpdateEcShards
looked. The 0/0-means-not-reported rule is unchanged.

The count tests now assert that the stored pointer changes while a
previously obtained one still reads its old values, and a -race test
runs full and incremental count refreshes against readers; it failed
on the previous code and passes now.
2026-11-03 12:48:16 -08:00
Chris LuandClaude Opus 5.5 5063aa9be2 weed-vector: check master certificates against the server allowlists, read -config_dir (#12374)
* weed-vector: look for security.toml in -config_dir first

weed processes search the directory -config_dir names before ., so a
cluster keeping security.toml there left the engine in plaintext.
--config-dir (also -config_dir, WEED_VECTOR_CONFIG_DIR) does the same.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* weed-vector: dial the masters the way weed clients do

The master's certificate must chain to grpc.ca and is not checked against
the host dialed, as weed clients verify (advancedtls.CertVerification), so
masters sharing one workload certificate, or dialed by IP, register as
they do from weed. Instead, grpc.client.master_allowed_server_commonNames,
grpc.client.allowed_server_commonNames or grpc.allowed_server_commonNames
(and the wildcard domain likewise) restrict which server identities are
accepted: the common name or a DNS SAN, or an exact URI SAN, as
security.Authenticator.AuthenticateServer does. These used to be refused
at startup; --tls-server-name is gone with the host check.

The TLS runs in a connector that reads the credentials asynchronously at
every dial, so the connect timeout still covers the reads, the TCP connect
and the handshake together. An IPv6 master's brackets are stripped before
the dial.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test/s3vectors: the hardened leg allowlists server identities for grpc.client

The gateway and the engine then accept only servers named localhost, which
exercises the engine's allowed_server_commonNames check on the master dial.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* weed-vector: parse the refresh interval with time.ParseDuration's arithmetic

The interval was summed in floating-point seconds, so 0.9ns rounded up to
1ns where Go truncates it to 0 and falls back to the 5h default, 0.3s was
off by a nanosecond, and 3000000h was accepted where Go rejects anything
past the longest time.Duration. Each part is now an integer count of
nanoseconds plus the fraction Go adds, with Go's overflow bounds, and the
value is no longer trimmed, as Go does not trim it either.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* weed-vector: test a master dial whose credential read stalls

The bounded-dial tests passed no credentials or a prebuilt endpoint, so
moving the credential read out of the connect timeout would have failed
none of them. The new test points grpc.ca at a FIFO nobody writes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* weed-vector: dial a closed port in the stalled credential read test

Against a silent listener, a dial that never read the credentials timed
out with the same message, so the test passed without the read. Nothing
listens on the port now, so only the stalled read can time out.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* weed-vector: test that Security::load searches the config dir

Only candidates() was tested, so load dropping config_dir on the way
would have passed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* weed-vector: expand a leading ~ in -config_dir like weed's ResolvePath

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* weed-vector: hold the port in the stalled credential read test

A released port can be taken before the dial, and a dial that skipped the
read would then time out and pass. A listener that drops every connection
keeps the port and still fails such a dial at once.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
2026-11-03 12:41:02 -08:00
Chris Lu 30f0364256 admin: fail an in-flight detection when its worker disconnects (#12376)
* admin: keep a reconnected plugin worker registered when its old stream unwinds

cleanupSession removed whatever session was registered under the worker
ID, so a worker that reconnected while its previous stream was still
unwinding lost the new session and its registry entry to the old
stream's deferred cleanup. Tie cleanup to the stream that is ending and
publish the new stream before the registry upsert so a lingering
predecessor sees itself superseded.

* admin: fail an in-flight detection when its worker disconnects

RunDetectionWithReport waited only on the caller's context and the
DetectionComplete channel, so a detection whose worker stream ended
mid-request could only finish by timeout. On a customer cluster a
planned restart of ec-vacuum-1 during an ec_vacuum detection left the
default lane in 'detecting' with every execution slot idle until the
3600s detection timeout, blocking every job type in that lane.

Bind the request to the stream it was sent on and fail the detection
as soon as that stream closes, whether by disconnect or by a reconnect
that replaced the session. A liveness tick also fails it once the
registry no longer considers the worker available (heartbeat stale),
sending a cancel in that case because the stream may still be open.
The detector lease is dropped so the next pick starts fresh, and a
'disconnected' detector activity is recorded. WorkerDisconnectForTest
now goes through the real session cleanup.

* admin: retry a scheduled detection once when its detector drops

When a detection fails because its worker disconnected, pick a detector
again and retry once within the same iteration while the job type's
runtime budget allows, instead of giving the whole cycle up. Any other
failure, or a failed retry, ends the run as before and the lane moves
on; the next cycle re-detects anyway.

* admin: register a plugin worker's hello and stream as one
 step

WorkerStream published the stream under sessionsMu but ran the registry
upsert after releasing it, so two concurrent streams for one worker ID
could publish their hellos in the opposite order from their sessions:
the registry then described the older hello's capabilities while only
the newer stream was sendable, and detector picks routed on them.

registerSession upserts the hello and installs the stream under one
sessionsMu critical section, and a hello or heartbeat arriving on a
stream that is no longer the registered one is dropped instead of
rewriting the live stream's entry.

* admin: steer a detection retry off the stream that dropped

The retry picked while the lost worker could still be in the registry:
cleanupSession closed done before taking the registry entry down, and a
stream closed by a reconnect keeps its entry on purpose. The lost worker
ranks first on free slots, so the single retry went back to it and
failed on send while a healthy worker sat idle.

The disconnect error now carries the stream it came from and the retry
picks around it; pickDetector only returns workers whose current stream
is open, excluding registry entries whose stream is closed; and
cleanupSession removes the registry entry before closing done, so a
waiter woken by the closure never sees the dead worker as a candidate.
2026-11-03 12:28:39 -08:00
Chris LuandClaude Opus 5.5 eeb8aefe70 weed-vector: gRPC TLS to the masters and for the gateway (#12368)
* weed-vector: read the cluster's security.toml

The engine finds security.toml where weed processes do (., ~/.seaweedfs,
/usr/local/etc/seaweedfs, /etc/seaweedfs) with WEED_* environment
variables over it, and builds gRPC client and server credentials by the
rules of security.LoadClientTLS and LoadServerTLS: a dedicated client
certificate is preferred, both halves of a pair must be set, and grpc.ca
is required once either is.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* weed-vector: gRPC TLS to the masters and from the gateway

The masters are dialed with [grpc.client]'s credentials when the cluster
has them, the master's certificate checked against the host dialed or
--tls-server-name. The engine's own listener takes its identity from
[grpc.vector] and then requires a client certificate chaining to grpc.ca,
which is how the gateway, dialing with [grpc.client], reaches it. The
rustls provider is pinned before the first TLS use, as the other binaries
linking both aws-lc-rs and ring do.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* scaffold: document [grpc.vector]

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* test/s3vectors: put every gRPC hop under mutual TLS in the hardened leg

VECTOR_TLS=1 now also writes [grpc.master], [grpc.volume], [grpc.filer],
[grpc.s3], [grpc.client] and [grpc.vector], and probes the master's and
the engine's gRPC ports: plaintext HTTP/2 and a client without a
certificate are refused, the client certificate gets a SETTINGS frame.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* test/s3vectors: give the test volume server 64 volume slots

Every collection grows 7 volumes and test_vector_bucket_encryption opens
two buckets while the previous test's bucket is still being torn down, so
16 slots were marginal: the Go legs failed CreateIndex now and then with
"failed to find writable volumes" and master.log showing "create 7
volume, created 4: Not enough data nodes found!".

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* weed-vector: reload the listener's certificates and enforce allowed_commonNames

The listener now runs the way weed servers run theirs. The certificate, key
and CA files are re-read every WEED_TLS_CERT_REFRESH_INTERVAL (weed's
default of 5h) and on SIGHUP, and new handshakes use the changed files
without a restart; a file that no longer parses is logged and the working
configuration stays. [grpc.vector] allowed_commonNames and
grpc.allowed_wildcard_domain restrict which client certificates are served,
checked against the subject common name as security.Authenticator and the
Rust volume server's CommonNameVerifier do.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* scaffold: allowed_commonNames and the refresh cadence for [grpc.vector]

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* test/s3vectors: the engine's allowlist keeps an outsider certificate out

The hardened leg sets [grpc.vector] allowed_commonNames = "localhost" and
probes the engine with a second certificate the same CA signed for
CN=outsider: the handshake must be refused, while the gateway's certificate
still gets an HTTP/2 SETTINGS frame.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* weed-vector: take the master's TLS name from the parsed endpoint

A master given without a port (https://master.example) had the whole URL
checked against its certificate.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* weed-vector: bound the master TLS handshake

connect_timeout covers only the TCP connect, so a master that accepted the
connection and never answered the handshake held up the other masters.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* weed-vector: look up security.toml keys the way viper does

An empty WEED_* variable no longer hides the file's value, and keys match
regardless of case, as for weed processes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* weed-vector: refuse master allowlists it cannot enforce

weed checks a dialed server against allowed_server_commonNames and
allowed_server_wildcard_domain on top of grpc.ca. The engine's master dial
accepted every certificate the CA signed even when one was set; it now
fails at startup and names the key.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test/s3vectors: read the whole frame header in the gRPC TLS probe

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test/s3vectors: probe the volume, filer and S3 gRPC listeners too

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* weed-vector: read the master dial credentials only when given masters

A standalone engine never dials with grpc.client, so its settings no longer
stop it from starting.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* weed-vector: run each listener TLS handshake on its own task

The accept loop awaited every handshake before accepting the next
connection, so a peer that connected and sent nothing stopped every
gateway connection after it. Handshakes now run concurrently, each under a
10s timeout.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* weed-vector: read the master dial's TLS files at every dial

The client certificate, key and CA were read once at startup, so after the
files rotated a reconnect still presented the old pair. Each dial now reads
them, as the listener re-reads its own.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* weed-vector: read every unit Go durations accept in the refresh interval

WEED_TLS_CERT_REFRESH_INTERVAL values in ns, us or a leading + fell back
to five hours in the engine while weed servers honored them.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* weed-vector: fall back on a refresh interval too long to hold

An oversized WEED_TLS_CERT_REFRESH_INTERVAL panicked the engine at
startup; weed servers reject it and keep the default.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* weed-vector: refuse allowed_commonNames without the listener's certificate

With [grpc.vector] allowed_commonNames set but no cert and key, the
listener served plaintext and checked no client at all.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* weed-vector: read the master dial's credentials inside its connect timeout

The CA, certificate and key were read synchronously before the timeout
started, so a stalled credential path held up every other master.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* weed-vector: let a set key win over an empty spelling of it

With allowed_server_commonNames = "" beside allowed_server_commonnames
= "master", the lookup stopped at the empty exact match and the
allowlist went unnoticed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
2026-11-03 11:42:49 -08:00
Chris LuandClaude Opus 5.5 2ec0623073 filer UI, admin: upload a folder (#12371)
* admin: place uploaded files by their path inside an uploaded folder

The multipart file name keeps only the base name, so the upload API takes
an optional relativePaths field listing each file's path under the target
directory. Paths that are empty, absolute or contain . or .. components
are rejected per file.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* admin: upload a folder from the file browser

The upload modal gets a folder picker, and a folder dropped on the listing
is walked into, so its files upload with their subfolder paths.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* filer UI: upload a folder

An Upload Folder button picks a whole folder, and a folder dropped on the
listing is walked into. Each file is posted to its subfolder under the
current directory, which the filer creates on the first upload into it.
Progress rows are keyed by that path, so equal names in different
subfolders no longer share a row.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* admin: reject an empty relativePaths field instead of ignoring it

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* admin: upload a large selection in batches below the multipart part limit

Go's multipart parser rejects a form of more than 1000 parts, so a folder
of about a thousand files failed as a whole. The upload modal now sends
500 files per request and sums the results; progress covers all batches,
and the button stays disabled until the last one finishes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* admin: escape file names in upload error messages

A failed upload reports the client-supplied file or folder path, and the
toast renders its message as HTML.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* admin: list failed uploads in the modal and reload the listing when it closes

A partial upload logged its failures only to the console before the page
reloaded, and a batch that failed after earlier ones succeeded left the
listing stale. The failures now stay in the open modal, and closing it
reloads the listing once any file was uploaded.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* admin: cancel the upload when its modal closes

Closing the modal, including with its Cancel button, left the batch loop
running, so the remaining batches were still written. The batch in flight
is now aborted, which also stops the loop, and the listing reloads since
the aborted request may already have stored files.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* admin: count upload progress by files when the selection has no bytes

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* admin: keep the newest upload selection when a folder drop finishes late

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* admin: cap the upload preview at a fixed number of rows

A folder of many thousands of files rendered one preview row per file.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* filer UI: upload a few files at a time

Every selected file started its own request at once, so a large folder
opened thousands of requests together.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* admin: regenerate the gzipped admin.js

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* admin: reload the listing after a batch fails without a result

A batch that fails mid-request may still have stored some of its files,
so closing the modal reloads the listing.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* admin: stop reading a folder drop once a newer selection replaces it

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* admin: skip the read error of a folder drop a newer selection replaced

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* filer UI: share the four-upload limit across selections

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* admin: regenerate the gzipped admin.js

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* admin: set up the file browser handlers once

admin.js already runs setupFileManagerEventHandlers on DOMContentLoaded,
so the file browser page registered every listener twice and each drop
was read twice.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-03 11:35:32 -08:00
Chris LuandClaude Opus 5.5 d5e359d0e2 rust volume: run each gRPC TLS handshake on its own task (#12372)
* rust volume: run each gRPC TLS handshake on its own task

The incoming stream awaited every handshake before accepting the next
connection, so a peer that connected to a TLS gRPC port and sent nothing
stopped every gRPC connection after it. Handshakes now run concurrently,
each bounded by grpc-go's default 120s connection timeout, as a Go server's
are.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* rust volume: retry a failed gRPC accept instead of closing the listener

grpc-go backs off and retries temporary accept errors such as EMFILE.
The TLS incoming stream ended on the first accept error, which stopped
the gRPC port for the rest of the process lifetime.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* rust volume: test that the gRPC handshake timeout closes a silent peer

The handshake timeout becomes a parameter so a test can use a short one.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-03 11:23:07 -08:00
Chris LuandClaude Opus 5.5 3b7706bad8 rust volume: exit cleanly on SIGTERM, and make the harness notice when a server does not (#12370)
* rust volume: exit cleanly on SIGTERM and SIGINT

The shutdown path raced the HTTP and gRPC tasks with select!, then
awaited both handles again; tokio panics when a JoinHandle is polled after
it completed, so every signal ended in "JoinHandle polled after
completion" with exit status 101 and the volumes never closed. Waiting
for both tasks with join! is enough: both stop on the shutdown signal, and
a gRPC serve error sends it.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* test/volume_server: fail when a stopped server does not exit cleanly

The harness ignored the exit status after SIGINT, which is how the Rust
volume server's shutdown panic went unseen. Every server it stops must
now exit 0 within 10s. The Go volume servers get -preStopSeconds=0, which
the Rust ones already had, so a stop does not spend the default 10s grace
inside that window.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* test/volume_server: let assertArgPair find flags after a bare flag

rustVolumeArgs carries --volume.allowUntrustedRemoteEndpoints without a
value, so pairs no longer sit at even indexes and
TestRustVolumeArgsIncludeReadMode reported --readMode missing although it
was there.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* rust volume: stop both servers when either one exits

A gRPC task that panicked left the HTTP server and heartbeat running
until a signal arrived. Whichever server task ends first now sends the
shutdown and the other is awaited once, matching Go where a serve
failure is fatal.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* rust volume: report an HTTP task panic as a server error

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
2026-11-03 10:44:01 -08:00
Chris LuandClaude Opus 5.5 cadb614630 filer.sync: stop chunk retries with the same supersession check the skip uses (#12367)
* filer.sync: pass the replayed entry down to the chunk retry gates

The fetch, manifest-resolve, and manifest-upload retry gates only received the
replayed version's mtime, which is not enough to order it against the live
source entry. Pass the entry itself; the gates still compare mtime only, so
behavior is unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* filer.sync: stop chunk retries with the same supersession check the skip uses

The fetch, manifest-resolve, and manifest-upload retry gates stopped early only
for a gone or strictly newer source entry. A transient error therefore kept
retrying a version the source had rewritten under an older mtime until the
read recovered, while a live lookup that lags the replay could stop the
retries on its newer mtime alone. The error handler that runs once a gate stops
already decides with sourceMovedOn, which orders by commit timestamp and
compares chunk sets; the gates now use it too, and hasSourceNewerVersion has no
callers left.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* filer.sync: a lagging live lookup cannot prove a chunk dropped

A peer filer's event reaches subscribers before the source store applies it,
so the live lookup can return the version the replay replaces. That entry
never listed the replay's new chunk, so sourceDroppedChunk read a 404 on it as
an obsolete version and cut the missing-chunk wait short. Like sourceVersion,
it now ignores a live entry committed before the replayed version.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* filer.sync: test that a replicate failure on a superseded version is skipped

Covers onReplicateChunkError with a transient error for a deleted source entry,
a later commit with an older mtime, and an unchanged source, and points the
unconfirmed-supersession test comment at it instead of the helper test.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* filer.sync: test the manifest gate against a missing volume and a lagging lookup

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-03 10:43:22 -08:00
Chris LuandClaude Opus 5.5 8461ae2fe9 filer.sync: skip a rewritten source version regardless of its mtime order (#12366)
* filer.sync: a chunk the live source entry dropped is gone whatever its mtime says

sourceDroppedChunk refused to judge a 404'd chunk when the live source entry
carried an older mtime than the replayed version, treating that as a rollback.
Timestamp-preserving writers (cp -p, rsync -a, tar extracts) produce exactly
that order on every rewrite, so the vacuumed chunk sat in the retry loop for
the full missingSourceChunkGrace before being written off. Whether the live
entry still lists the chunk is independent of mtime order.

* filer.sync: treat a rewritten source entry as moved on regardless of its mtime

sourceVersion only compared chunk sets when the live entry was at least as
new as the replayed one, so a live entry stamped with an older mtime but
holding different chunks was never recognized as the current version. The
sink then fetched the replayed chunks, hit 404 on each vacuumed one, and the
same mtime guard in onReplicateChunkError kept the event from being skipped.
Whether the source rewrote the file is a chunk-set question, not an mtime
one; the live version is synced in its place as before.

* filer.sync: apply a substituted live snapshot even under a newer destination mtime

After sourceVersion swapped in the live entry, UpdateEntry re-ran
chooseUpdateAction on it and a destination whose mtime was above the live
entry's read as a late update, so the current source content was never
written. The live snapshot is the source's current state, not a replay, so
mtime order says nothing about it; repair from it unless the destination
already holds it.

* filer.sync: a live entry committed before the replay does not supersede it

Dropping the mtime guard let any chunk-set difference read as a rewrite,
including a live lookup that lags the event: a peer filer's event reaches
subscribers before the local store applies it, so the lookup can still
return the version the replay replaces. sourceVersion then synced that
older version over the destination and consumed the event. The meta-log
commit stamps order the two versions where mtime cannot, so a live entry
committed before the replay proves nothing.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* filer.sync: check commit order before the mtime supersession

A lagging live entry can carry a newer mtime than the replay, as when the
replayed version was stamped back to an older time, so sourceSupersedes
took it as newer and the older version was synced in place of the replay.
Rule out a live entry committed before the replay first.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-03 09:36:04 -08:00
Chris LuandClaude Opus 5.5 5c99744e00 rust volume: count sorted-map files and deletes the way Go does (#12365)
* rust volume: count sorted-map files and deletes the way Go does

The Rust sorted needle map still used the counter rule Go dropped when
the index loaders were unified: every .idx row, tombstones included,
counted as a file, and each delete added one file and two deletions.
Go counts put rows as files and a delete as one deletion of the live
row's bytes, so the Go and Rust copies of one volume reported different
FileCount and DeleteCount for the same index.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* rust volume: test canDelete volume counters across restarts

Mirrors the Go store test: a canDelete volume reloaded from .sdx must
count its deletes, and the counters it reports must match what the next
reload computes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-03 08:13:18 -08:00
5df7faf0e3 fix(storage): update deletion metrics in SortedFileNeedleMap.Delete (#12363) (#12364)
* fix(storage): update deletion metrics in SortedFileNeedleMap.Delete (#12363)

* storage: test canDelete volume counters across restarts

A canDelete volume reloaded from .sdx must count its deletes so
volume.deleteEmpty and vacuum see it drain, and the counters it reports
must match what the next reload computes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-03 08:05:45 -08:00
Chris LuandClaude Opus 5.5 d13d975b6e shell: keep fs.mergeVolumes and fs.distributeChunks from reverting entries written while they run (#12362)
* shell: keep fs.mergeVolumes from reverting entries written during the merge

The walk reads a page of entries and moves their chunks one at a time, then
wrote each entry back unconditionally. A file overwritten in between got its
old version back, and the filer deleted the new version's chunks as garbage,
so an acknowledged write was lost without any error.

The update now carries IF_ENTRY_EQUAL over the entry as the walk read it. When
the filer refuses because the entry changed or was deleted, the copies made on
the target volumes are deleted, their plan room is released, and the entry is
skipped; the source needles stay where the filer still points.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* shell: keep fs.distributeChunks from reverting a file overwritten while it runs

The command looks the entry up once, moves its chunks, and then wrote that
entry back unconditionally, so an overwrite during the redistribution was
reverted and the filer deleted the new version's chunks as garbage.

The update now carries IF_ENTRY_EQUAL over the looked-up entry. When the filer
refuses because the file changed or was deleted, the moved copies and any new
manifest chunks are deleted and the command reports it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* shell: keep s3.versions.audit -heal from clearing a pointer that just moved

The heal cleared the latest-version pointer on the directory entry the walk
read, unconditionally. A version written between the audit and the heal has
already pointed the directory at a live file, and clearing that pointer hid it.
The heal now carries IF_ENTRY_EQUAL over the audited entry and reports a
refusal as a failed heal.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* shell: keep fs.meta.changeVolumeId from reverting entries written meanwhile

The walk rewrote each entry's volume ids and wrote back the copy it had read,
unconditionally, so an overwrite between the read and the write was reverted
and its chunks were deleted as garbage. The update now carries IF_ENTRY_EQUAL
over the entry as read, and a refused entry is skipped and reported.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* shell: keep the target copies another fs.mergeVolumes run committed

moveChunk keeps the key and cookie, so two overlapping merges of the same
source into the same target write the same fid. When one run commits, the
other's update is refused, and deleting its copies deleted the needle the
committed entry now points at.

A refused update now re-reads the entry, resolving its manifests, and deletes
only the copies it no longer references. If the entry cannot be read, the
copies are left for volume.fsck.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* shell: let a later hard-link sibling retry a merge the filer refused

The first sibling of a HardLinkId claims the move for all of them. When its
update is refused, the claim made every later sibling skip, so the shared
chunks stayed on the source volume for the rest of the run. A refused update
now releases the claim, and a sibling listed after the change retries the move
against the entry as it is now.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* shell: delete the manifest blobs a failed fold in fs.distributeChunks saved

When re-manifestizing failed partway, the blobs already uploaded were only
counted in a log line and stayed on their volumes with nothing referencing
them. They have freshly assigned fids, so they are now deleted with the
cookie-checked delete the other orphan cleanups use.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* shell: check the heal clears the pointer file name too

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* shell: require the shell lock for fs.mergeVolumes -apply

moveChunk keeps the key and cookie, so two overlapping runs that move the same
chunk into the same volume write the same fid. Re-reading the entry before
deleting a refused run's copies covers the run that already committed, but not
one that commits after that read. With the lock, two applying runs can no
longer overlap, as volume.fsck and volume.merge already require. A dry run
still works without it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* shell: check unreferencedTargets reads the refused entry itself

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* shell: check unreferencedTargets keeps a copy a manifest references

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* shell: check unreferencedTargets keeps the manifest copy and skips an unreadable manifest

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-03 07:59:18 -08:00
76198d15cf fix(fsck): inspect references to deleted volumes (#12357)
* fix(fsck): inspect references to deleted volumes

* volume.fsck: treat every match-all collection filter as global

ALL_COLLECTIONS and EACH_COLLECTION compile to the same match-all
matcher as an empty filter or "*", but the string check treated them
as a restricted scope: a selected missing volume was rejected and
references to master-absent volumes were reported without being purged.
Use the compiled matcher instead.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* volume.fsck: purge an entry whose manifest is on a missing volume

A top-level chunk on a missing volume already proves the entry is
unreadable, whether or not its manifests resolve. Skipping such entries
kept the very case where the manifest needle sat on the retired volume,
and the remaining unresolved manifest then blocked every later orphan
purge. The missing-needle check on live volumes never had this gate, so
both paths now treat these entries the same. The warning now says only
orphan purging is disabled, which is the only thing the counter gates.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* volume.fsck: report a failed Filer delete

A rejected purge DELETE printed nothing unless -v was set, so the run
looked successful after "deleting path" while the entry stayed. The
filer answers 204 for an entry that is already gone, so only real
failures print.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* volume.fsck: tighten help text, comments and missing-volume tests

The help now wraps like the rest of it and says that a purge removes
the whole entry, including its chunks on other volumes. The live test
header no longer calls the run report-only, and the reverse-scan test
checks the report line for each missing chunk.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* volume.fsck: drop the missing-volume design notes

The notes said no destructive change was approved while the change adds
a confirmed purge, still described the dropped manifest gate, and
mostly recorded local runs and a reverse index that is not part of
this change. The operator contract lives in help volume.fsck.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* volume.fsck: allow a report-only missing-volume scan with -collection

The collection check exists to stop a purge that cannot be scoped, but
it also rejected a report-only run that deletes nothing. Apply it only
with -reallyDeleteFilerEntries, and limit the -confirmMissingVolumeDeletion
description to the -volumeId case it guards.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* volume.fsck: count only a 2xx answer as a purged entry

A redirect the client could not follow left the entry in place but was
recorded as deleted, which also queued its parent for empty-directory
cleanup.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* volume.fsck: say that a filtered purge of a missing -volumeId is refused

The help and the early warning both said a purge with -collection only
reports references to unlisted volumes, but a -volumeId naming one is
rejected before the scan. The help now says so, and the warning prints
after that check, so a refused run no longer claims it will report.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: MZ01-CHOIMINSU <mz01-juwannam@MZ01-CHOIMINSU.local>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-03 07:57:43 -08:00
Chris LuandClaude Opus 5.5 e8b284fef5 admin: scope plugin job and activity fetches to the selected job type (#12360)
* admin: scope plugin job and activity fetches to the selected job type

The plugin page fetched the newest 500 jobs and 500 activities
cluster-wide and filtered them by job type in the browser. On a busy
cluster that window can hold none of the selected type (433
erasure_coding + 64 ec_balance jobs, 497 ec_balance activities), so
/plugin/lanes/default/execution?job=ec_vacuum showed "No execution
jobs tracked yet" while the server had seven recent ec_vacuum jobs.

When a job-type tab is active, fetch the queue/detection/execution
lists with job_type=<selected> so the server filters before applying
the limit, and refetch when the selected job type changes. The
unscoped lists are still fetched for the status cards and the overview
per-type summary.

* admin: drop a plugin monitor refresh whose job type is no longer selected

A refresh started on one job-type tab could finish after the user moved
to another tab and that tab's refresh had already landed. Its rows, all
of the earlier job type, then replaced the current tab's and the
renderers filtered them down to empty tables until the next refresh.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* admin: show the lane-wide rows when a job-type scoped fetch fails

On a job-type tab a failed scoped jobs or activities request threw before
anything was stored, so the status cards, scheduler state and tables all
stayed stale. Fall back to the unscoped rows, which the renderers already
filter by job type.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* admin: filter plugin jobs and activities by lane before the limit

/api/plugin/jobs and /api/plugin/activities took the newest N rows across
every lane and then dropped the other lanes' rows, so on a busy default
lane a quieter lane's page could come back empty.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* admin: keep a failed plugin monitor fetch's last rows and report it

Falling back to the lane-wide rows when a scoped fetch failed could still
blank the tables, since that window may hold none of the selected type,
and the failure went unreported. A lane-wide or scheduler-state failure
also threw before the scoped rows that did arrive were stored.

Each request now keeps its previous value on failure: the scoped rows
only while the tab is unchanged, otherwise the lane-wide rows. Whatever
arrived is stored and rendered, then the first failure is rethrown so the
caller reports it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* admin: report a monitor refresh failure apart from the action it follows

expireJob and runWorkflow refreshed the monitor inside the action's try
block, so a failed refresh after a successful expire or workflow run was
reported as a failed action, and the expired job's detail was not
reloaded. Catch the refresh separately, as runDetection already does.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* admin: show a repeated plugin page refresh failure once

refreshAll runs every 3 seconds and notify opens a modal, so a request
that kept failing reopened the same alert on every tick. Report an error
only when it differs from the last one, and forget it once a refresh
succeeds.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-03 06:32:49 -08:00
Chris LuandClaude Opus 5.5 222cfa8e2e admin: summarize the EC delete_count accounting warning per refresh (#12359)
* admin: summarize the EC delete_count accounting warning per refresh

collectCollectionStats runs on every admin collection, bucket and
topology refresh, and it logged a Warning for each EC volume whose
delete_count summed across shard holders exceeds its file_count. That
sum is an expected accounting artefact (a tombstone can be recorded on
more than one holder), it recurs identically on every pass, and on one
cluster with hundreds of such volumes it grew the admin log to 20 GB,
more than 99% of it this one line.

Log one V(1) summary per pass with the affected volume count and keep
the per-volume detail at V(2). The object count is still skipped for
those volumes.

* admin: test that an EC volume with deletes above file_count is skipped

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-03 06:31:15 -08:00
Chris LuandClaude Opus 5.5 e5bc658eb7 test/s3vectors: a hardened cluster with TLS volume servers, read JWTs and encrypted chunks (#12356)
* test/s3vectors: a hardened cluster, with TLS volume servers, read JWTs and encrypted chunks

Three switches on the test cluster: VECTOR_TLS=1 puts the volume servers
behind HTTPS requiring client certificates and enables https.client for
every process; VECTOR_READ_JWT=1 gives every process a jwt.signing.read.key,
so volume reads need a token; VECTOR_ENCRYPT_VOLUMES=1 starts the gateway
with -encryptVolumeData. The integration matrix gains a hardened leg per
volume server with all three on.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* test/s3vectors: the hardened cluster proves its enforcement

Once the gateway is up, the harness checks that the hardening holds, not
only that it is configured: a TLS volume server refuses plain HTTP (Go
with HTTP 400, Rust with a TLS alert) and a connection without the client
certificate, and answers with it; under a read key an unsigned read of a
real chunk, written through the gateway and located through the filer,
gets 401.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* test/s3vectors: the hardened cluster proves its chunks are encrypted

With VECTOR_ENCRYPT_VOLUMES the probe reads the stored chunk straight
from the volume server, signing the read when a read key is set, and
requires a recorded cipher key and bytes that differ from the plaintext.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test/s3vectors: the signed probe read must return the chunk

An empty 200 passed both branches; an unencrypted read now has to return
the bytes written, and an encrypted one has to return something else.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test/s3vectors: a read key in the caller's environment does not leak into the cluster

Only VECTOR_READ_JWT turns read JWTs on; an inherited
WEED_JWT_SIGNING_READ_KEY made the volume servers refuse the probe's
unsigned read in an encrypt-only run.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test/s3vectors: quote certificate paths as TOML strings in security.toml

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test/s3vectors: keep non-ASCII certificate paths as UTF-8 in security.toml

TOML has no surrogate-pair escapes, and the Rust volume server reads the
quoted value without decoding escapes at all.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test/s3vectors: an encrypted probe chunk must not be a readable gzip stream

The gateway may gzip the body before encrypting it, so a chunk uploaded
compressed but unencrypted would differ from the body and still pass.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
2026-11-03 04:56:15 -08:00
Chris LuandClaude Opus 5.5 af45ab8f2a s3api: known-answer fixture generator for SSE-S3 and filer-encrypted chunks (#12355)
* s3api: a known-answer fixture generator for SSE-S3 and filer-encrypted chunks

TestGenerateDirectReadFixture writes, from this package's own encryption,
the pieces a reader outside the gateway must handle: SSE-S3 chunks with
their per-chunk metadata, the object-level key, a single-stream ciphertext,
the KEK derived from s3.sse.key and wrapped under a passphrase in both
on-disk formats, and chunks encrypted by the filer, plain and gzipped. It
is skipped unless DIRECT_READ_FIXTURE names the output file.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* s3api: the fixture generator restores the key manager and checks the write

The test set the global key manager's KEK and left it; a later test in
the same run then found a signing key where it expects none. The previous
key is restored with t.Cleanup. The fixture file's close error is checked
too, so a delayed write failure fails the test instead of leaving a short
fixture.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* s3api: the fixture generator unwraps both KEK formats before writing

The legacy v1 wrap is built by hand in the test; reading v1 and v2 back
through unwrapKEK ties both to the format the key manager accepts.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3api: pin a legacy v1 wrapped KEK made outside the package

Generating the v1 sample through legacyKEKWrappingSalt let a change to
the salt pass on both sides; a vector wrapped elsewhere under the
historical salt must still unwrap.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
2026-11-03 04:53:57 -08:00
199b2e2dd9 fix(s3api): reject an invalid storage class before uploading the body (#12354)
* fix(s3api): reject an invalid storage class before uploading the body

* fix(s3api): reject an invalid storage class on CreateMultipartUpload

* fix(s3api): validate storage class and tags before reading the PUT body

* fix(s3api): validate storage class and tags with the other PUT headers

Checking them next to ValidateRequestEncryption keeps the check ahead of
any later hook that may answer the PUT without reaching putToFiler.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* refactor(s3api): reuse validatePutHeaders in putToFiler

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(s3api): store storage class and tags on directory-marker PUTs

The marker path validated x-amz-storage-class and x-amz-tagging but its
mkdir callback never wrote them, so a valid value was accepted and then
dropped and GetObjectTagging on the marker came back empty. Store both
the way putToFiler does.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test(s3api): read directory-marker tags back through GetObjectTagging

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-02 22:08:21 -08:00
Chris LuandClaude Opus 5.5 c2ca83c0bf filer: say jwt.filer_signing.key gates only the IAM gRPC service (#12353)
* filer: say jwt.filer_signing.key gates only the IAM gRPC service

The key never covered the rest of the filer gRPC port, which can read and
write the IAM store under /etc/iam. Point operators at [grpc.filer] mTLS for a
port reachable beyond a trusted network instead of the key.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* filer: note the IAM gRPC OIDC provider and role RPCs are refused without the key

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-02 18:00:59 -08:00
github-actions[bot] cc71cc9585 docs: regenerate star history chart 2026-11-03 01:08:52 +00:00
Chris LuandClaude Opus 5.5 9b45aae2eb operation: pad the marginal gzip test past Go 1.26's stored block (#12352)
At BestSpeed, Go 1.26's flate writes 4000 random bytes plus a 300-byte
zero run as a stored block, so the gzip output was 28 bytes larger than
the input and the test failed its own precondition on the toolchain
go.mod pins. With 450 zero bytes the saving is 5.7% on Go 1.26 and 8.4%
on Go 1.27, inside the marginal range on both.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-02 17:01:40 -08:00
dependabot[bot] 794cebe7b0 build(deps): bump cloud.google.com/go/kms from 1.35.0 to 1.36.1 (#12342)
Bumps [cloud.google.com/go/kms](https://github.com/googleapis/google-cloud-go) from 1.35.0 to 1.36.1.
- [Release notes](https://github.com/googleapis/google-cloud-go/releases)
- [Changelog](https://github.com/googleapis/google-cloud-go/blob/main/documentai/CHANGES.md)
- [Commits](https://github.com/googleapis/google-cloud-go/compare/kms/v1.35.0...dlp/v1.36.1)

---
updated-dependencies:
- dependency-name: cloud.google.com/go/kms
  dependency-version: 1.36.1
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-11-02 13:01:04 -08:00
Chris LuandClaude Opus 5.5 8125a1549a s3vectors: support tag conditions in vector bucket policies (#12351)
* s3tables: resolve s3vectors:VectorBucketTag like s3tables:TableBucketTag

Both name a tag on the bucket that holds the target, whichever resource
the request acts on.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* s3vectors: give vector policy conditions the bucket, index and request tags

A vector bucket policy is evaluated with these condition keys:

- aws:ResourceTag/<key>: the tags of the bucket, or of the index when the
  request names one.
- s3vectors:VectorBucketTag/<key>: the bucket's tags, for bucket and
  index requests alike.
- aws:RequestTag/<key> and aws:TagKeys: the tags TagResource and
  CreateIndex set, and the keys UntagResource removes.

An index's tags are read only when the policy has an aws:ResourceTag
condition. The bucket's come from the lookup that already reads its
policy.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* s3vectors: accept tag conditions in vector bucket policies

PutVectorBucketPolicy rejected every Condition, since no condition key
was supplied. A condition is now accepted when its operator is one the
evaluator knows and each key is aws:ResourceTag/<key>,
aws:RequestTag/<key>, aws:TagKeys or s3vectors:VectorBucketTag/<key>.
Any other key would never match, so it is still rejected, as is a
condition field not spelled Condition.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* test/s3vectors: cover tag conditions in vector bucket policies

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* s3vectors: reject condition values the evaluator cannot compare

A null condition value was accepted and then panicked the request that
matched its statement, a Deny reaching even the bucket owner. A value
must now be a string, number or boolean, or a non-empty list of them.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-02 12:43:12 -08:00
dependabot[bot] 97e03f5a9f build(deps): bump actions/setup-go from 5 to 7 (#12347)
Bumps [actions/setup-go](https://github.com/actions/setup-go) from 5 to 7.
- [Release notes](https://github.com/actions/setup-go/releases)
- [Commits](https://github.com/actions/setup-go/compare/v5...v7)

---
updated-dependencies:
- dependency-name: actions/setup-go
  dependency-version: '7'
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-11-02 12:35:26 -08:00
dependabot[bot] b91de44615 build(deps): bump github.com/peterh/liner from 1.2.2 to 1.2.3 (#12341)
Bumps [github.com/peterh/liner](https://github.com/peterh/liner) from 1.2.2 to 1.2.3.
- [Commits](https://github.com/peterh/liner/compare/v1.2.2...v1.2.3)

---
updated-dependencies:
- dependency-name: github.com/peterh/liner
  dependency-version: 1.2.3
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-11-02 12:01:10 -08:00
dependabot[bot] 20c8876a52 build(deps): bump github.com/aws/aws-sdk-go-v2/service/sts from 1.51.1 to 1.51.6 (#12346)
build(deps): bump github.com/aws/aws-sdk-go-v2/service/sts

Bumps [github.com/aws/aws-sdk-go-v2/service/sts](https://github.com/aws/aws-sdk-go-v2) from 1.51.1 to 1.51.6.
- [Release notes](https://github.com/aws/aws-sdk-go-v2/releases)
- [Commits](https://github.com/aws/aws-sdk-go-v2/compare/service/s3/v1.51.1...service/sts/v1.51.6)

---
updated-dependencies:
- dependency-name: github.com/aws/aws-sdk-go-v2/service/sts
  dependency-version: 1.51.6
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-11-02 12:00:58 -08:00
dependabot[bot] a670553813 build(deps): bump google.golang.org/api from 0.297.0 to 0.310.0 (#12343)
Bumps [google.golang.org/api](https://github.com/googleapis/google-api-go-client) from 0.297.0 to 0.310.0.
- [Release notes](https://github.com/googleapis/google-api-go-client/releases)
- [Changelog](https://github.com/googleapis/google-api-go-client/blob/main/CHANGES.md)
- [Commits](https://github.com/googleapis/google-api-go-client/compare/v0.297.0...v0.310.0)

---
updated-dependencies:
- dependency-name: google.golang.org/api
  dependency-version: 0.310.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-11-02 12:00:36 -08:00
dependabot[bot] 9ae31fa425 build(deps): bump actions/setup-python from 5 to 7 (#12345)
Bumps [actions/setup-python](https://github.com/actions/setup-python) from 5 to 7.
- [Release notes](https://github.com/actions/setup-python/releases)
- [Commits](https://github.com/actions/setup-python/compare/v5...v7)

---
updated-dependencies:
- dependency-name: actions/setup-python
  dependency-version: '7'
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-11-02 12:00:20 -08:00
dependabot[bot] e166629f9b build(deps): bump go.etcd.io/etcd/client/pkg/v3 from 3.7.1 to 3.7.3 (#12344)
Bumps [go.etcd.io/etcd/client/pkg/v3](https://github.com/etcd-io/etcd) from 3.7.1 to 3.7.3.
- [Release notes](https://github.com/etcd-io/etcd/releases)
- [Commits](https://github.com/etcd-io/etcd/compare/v3.7.1...v3.7.3)

---
updated-dependencies:
- dependency-name: go.etcd.io/etcd/client/pkg/v3
  dependency-version: 3.7.3
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-11-02 11:59:59 -08:00
Chris LuandClaude Opus 5.5 b7b2a5947f admin: configurable file browser upload size limit (#12348)
* admin: carry a file browser upload size limit

The admin server keeps the total size limit of one file browser upload, 500 MB by default as the UI enforced before, and hands it to the file browser page.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* admin: reject file browser uploads over the size limit

The 500 MB limit was only checked in the browser, so any client could post larger uploads and fill the temp dir with spooled parts. Cap the request body at the limit plus room for the multipart framing, and answer 413 when the files add up to more than the limit.

Parse the form before reading the path field. FormValue used to parse it first with the default 32 MB memory, which made the later 1 GB ParseMultipartForm a no-op; keep 32 MB explicitly.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* admin: show and check the configured upload limit in the file browser

The upload dialog reads the limit from the page instead of a hard-coded 500 MB, lists it under the file picker, and skips the check when uploads are unlimited.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* admin: add -uploadMaxSizeMB to set the file browser upload limit

weed admin -uploadMaxSizeMB and weed mini -admin.uploadMaxSizeMB set the total size limit of one file browser upload; 0 removes it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* admin: note why the upload framing allowance is enough

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* admin: test an upload whose files only exceed the limit together

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* admin: remove spooled upload parts when the upload handler returns

Parts over 32 MB spool to temp files. The router passes the handler a copy of the request, so net/http never sees the parsed form and never removes those files. Every large upload, and every one rejected for size after parsing, left them on disk.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-02 11:32:03 -08:00
Chris LuandClaude Opus 5.5 a074538c8f admin: manage S3 vector buckets (#12334)
* s3vectors: run an operation for trusted local tooling

Like s3tables.Manager.Execute, Server.Execute runs one S3 Vectors operation
through its handler as a given identity, without a signed request. The admin
console can then manage vector buckets through the same validation,
authorization and catalog steps as the API.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* admin: list, inspect, create and delete vector buckets over the API

/api/s3vectors runs the S3 Vectors operations through Server.Execute as the
admin account: list vector buckets with their encryption, read one bucket
with its policy and each index's data type, dimension, distance metric and
non-filterable keys, create and delete a bucket, and delete an index. Going
through the operations keeps the API's rules, such as ICEBERG buckets not
being vector buckets and the implicit index namespace not blocking a delete.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* s3vectors: export the rule for which table buckets are vector buckets

IsVectorBucketFormat names the check ListVectorBuckets and isVectorBucket
already make, so the admin console can tell the same buckets apart.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* admin: vector bucket pages

Object Store > Vector Buckets lists the vector buckets with their ARN,
encryption and creation time, creates and deletes them, and opens one bucket
with its indexes (data type, dimension, distance metric, non-filterable keys)
and its policy, where an index can be deleted. A LANCE row on the Table
Buckets page links to the same bucket's vector page, since both APIs serve it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* admin: stop the vector bucket and index lists at the list limit

A page cut short by filtered ICEBERG buckets let the next page carry
the list past s3TablesAdminListLimit.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* admin: say when a vector bucket or index list is cut at the limit

The lists stop at s3TablesAdminListLimit while more remain, so mark
them truncated and point to an S3 Vectors client for the rest.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* admin: indent the vector bucket policy

A policy stored as one line ran off the right edge of the details page.
Policies that are not valid JSON are shown as stored.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* admin: show a deeply nested vector bucket policy as stored

Indenting multiplies the size of a deeply nested document, so a policy
past 32 levels of nesting is shown without indenting.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-02 11:25:24 -08:00
Chris LuandClaude Opus 5.5 0176231ea4 volume: replicas keep the primary's compression and metadata (#12350)
* volume(rust): replicate the stored needle instead of the request body

The primary forwarded the original request body and headers, so a
replica re-derived compression from the raw bytes. Send what Go's
ReplicatedWrite sends: a one-part multipart upload of the stored bytes,
gzip-marked when compressed, with the name, mime, pairs and Content-MD5,
and carry ts, ttl, cm and fsync in the query so form-field values reach
the replica too.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* volume(rust): keep the primary's compression on replica writes

A type=replicate upload now stores the bytes as received, like Go's
ParseUpload, so a replica no longer compresses data the primary stored
uncompressed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* volume(rust): replicas keep the primary's name and mime

Forward the primary's filename and mime even past the 255-byte needle
fields, so a replica applies the same limit and sets the same flags,
and skip the filename MIME guess on type=replicate, so an untyped
upload stays untyped on every copy.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* volume(rust): keep the multipart part Content-Type as sent

Parsing the part type as a mime value lowercased it and dropped values
it could not parse, so a replica stored a different mime than the
primary. Read the header verbatim, as Go's part.Header.Get does.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* volume: replicas keep the primary's mime as forwarded

A Go replica guessed the mime from the file extension, so an upload the
primary stored as application/octet-stream, or with no mime at all, got
a different type on the replica. On type=replicate keep the part's
Content-Type as sent, and have the replication sender leave an empty
mime empty instead of filling it in from the extension. This matches
what the Rust replica does.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* volume(rust): pick the JPEG orientation fix by file name, as Go does

Go checks the mime and the upload's file name, Rust checked the mime and
the URL path, so the same upload could be rotated on one server and not
on the other. With the name and mime forwarded as stored, a replica now
reaches the primary's decision, and a rotated image has no EXIF tag left
to act on, so the fix stays a no-op there.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-02 11:24:51 -08:00
Chris LuandClaude Opus 5.5 ec968aa016 s3vectors: support TagResource, UntagResource and ListTagsForResource (#12335)
* s3vectors: tag vector buckets and indexes

TagResource, UntagResource and ListTagsForResource answer on
/tags/{resourceArn} (POST, DELETE, GET) and by name. Each authorizes its
own s3vectors action on the bucket or index ARN, then reads or writes the
catalog record's tags. A resource holds at most 50 tags.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3vectors: apply the tags passed to CreateVectorBucket and CreateIndex

The tags field was accepted and dropped. It is now stored on the new
bucket or index, and creating with tags needs s3vectors:TagResource in
addition to the create action.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test/s3vectors: cover resource tagging end to end

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3tables: keep tag operations to a bound table bucket

TagResource, UntagResource and ListTagsForResource run through Execute
with WithTableBucketCreatedAt now answer not found when the bucket name
holds another bucket, as the bound creates and deletes already do.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3vectors: bind tagging to the authorized bucket and to vector indexes

The catalog tag calls carry the creation time of the bucket authorizeVector
approved, so a bucket created again under the name is not touched. An index
ARN must name a Lance table: another table in the vector namespace of a
format-less bucket is refused, as DeleteIndex does.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3vectors: match a TagResource grant scoped to the bucket being created

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3tables: merge tag writes into the tags the record holds at the write

TagResource and UntagResource merged into the tags they read before the
compare-and-swap and wrote that map on every attempt, so a tag write that
landed in between was undone. The merge now runs inside the mutation.
WithMaxTags lets an in-process caller cap the merged set there.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3vectors: check the 50-tag limit inside the catalog's tag write

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3tables: check a tag operation's bound bucket after its reads

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3vectors: validate each tag key UntagResource removes

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3vectors: validate untag keys one at a time

UntagResource sized a map by the number of submitted tag keys before
validating them, so a large body of repeated keys allocated far more than
the request itself, before authorization ran.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3vectors: authorize UntagResource before validating its keys

Validating a near-32 MiB body of tag keys costs about 13x its JSON
decode, so an unauthorized caller could make the gateway spend that
work on a request it then refuses.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-02 10:33:26 -08:00
Chris LuandClaude Opus 5.5 690aabda0e filer: reuse IsEqualOrUnder for the rename subdirectory check (#12338)
* filer: test the rename subdirectory check through CanRename

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* filer: reuse IsEqualOrUnder for the rename subdirectory check

A trailing slash on the old name made the source path end in a slash,
so a child of the source was no longer seen as under it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-02 10:28:48 -08:00
d650abe0e7 compression: compress JSON uploads, and match file types by extension (#12339)
* compression: compress JSON uploads, and match file types by extension

IsCompressableFileType did not recognise JSON media types, so S3 uploads of application/json (which carry no file name) were never compressed. Its callers also passed filepath.Base of the file name where it expects the extension, so the .json/.js/.html/... extension rules never matched a real file name. Both callers now pass filepath.Ext; the function lower-cases the extension and media type, ignores media-type parameters ("application/json; charset=utf-8"), and treats */json and *+json types as compressible.

* compression: treat application/json-seq as JSON

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* volume(rust): compress JSON uploads and match media types without parameters

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* operation: keep uploader gzip only when it saves at least 10%

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Pat White <5015863+patwhite@users.noreply.github.com>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-02 10:15:01 -08:00
Jiri PucandClaude Opus 5.5 1637941368 filer: allow renaming into a sibling that shares the name prefix (#12337)
CanRename rejected a move whenever the new parent directory's path started
with the source path as a plain string, so moving /dir/a into /dir/a.d
(or anything under /dir/ab) failed with "can not move directory to a
subdirectory of itself". Check the path boundary instead: reject only
when the new parent is the source itself or lies under it.


Claude-Session: https://claude.ai/code/session_01VncHKTehVroSAzRMUHykEX

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-02 10:09:22 -08:00
Chris LuandClaude Opus 5.5 7298b4ed37 fix(s3tables,s3vectors): bind DeleteNamespace and the vector bucket cleanup to the authorized records (#12331)
* s3tables: bind DeleteNamespace to the authorized namespace

The delete claimed whatever finished namespace held the name at claim
time, so one deleted and created again by another owner after the
authorization read was removed under the first one's authorization.
The claim now requires the creation time that was authorized.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3vectors: bind DeleteVectorBucket's namespace cleanup to the authorized bucket

After the bound DeleteTableBucket reported the bucket not empty, the
index namespace cleanup ran unbound, so a bucket deleted and created
again in between lost its empty index namespace under the first
bucket's authorization. DeleteNamespace now honors the bucket binding,
checking the bucket after reading the namespace, and the cleanup
carries it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-02 09:55:09 -08:00
811 changed files with 84231 additions and 8358 deletions

No files matched your search

+1 -1
View File
@@ -15,7 +15,7 @@ runs:
using: composite
steps:
- name: Install cosign
uses: sigstore/cosign-installer@v4.1.2
uses: sigstore/cosign-installer@v4.1.3
- name: Sign
shell: bash
+25 -1
View File
@@ -9,6 +9,8 @@ logs a warning rather than under-reporting.
import json
import os
import sys
import time
import urllib.error
import urllib.request
from datetime import datetime
@@ -22,6 +24,28 @@ REPO = "seaweedfs/seaweedfs"
TOKEN = os.environ["GITHUB_TOKEN"]
OUT = os.environ.get("OUT", "note/star_history.svg")
PAGE_CAP = 400 # GitHub's hard limit on stargazer pagination
REQUEST_ATTEMPTS = 5
RETRYABLE_STATUS_CODES = {429, 500, 502, 503, 504}
def open_url(req):
for attempt in range(REQUEST_ATTEMPTS):
try:
return urllib.request.urlopen(req, timeout=30)
except urllib.error.HTTPError as error:
if error.code not in RETRYABLE_STATUS_CODES or attempt == REQUEST_ATTEMPTS - 1:
raise
retry_after = error.headers.get("Retry-After")
try:
delay = min(float(retry_after), 60) if retry_after else 2**attempt
except ValueError:
delay = 2**attempt
error.close()
except urllib.error.URLError:
if attempt == REQUEST_ATTEMPTS - 1:
raise
delay = 2**attempt
time.sleep(delay)
def fetch_stargazers():
@@ -38,7 +62,7 @@ def fetch_stargazers():
"User-Agent": "seaweedfs-star-history",
},
)
with urllib.request.urlopen(req) as resp:
with open_url(req) as resp:
batch = json.load(resp)
if not batch:
break
+75
View File
@@ -0,0 +1,75 @@
import importlib.util
import os
from pathlib import Path
import sys
import types
import unittest
from unittest.mock import patch
from urllib.error import HTTPError
def load_star_history():
matplotlib = types.ModuleType("matplotlib")
matplotlib.use = lambda backend: None
pyplot = types.ModuleType("matplotlib.pyplot")
dates = types.ModuleType("matplotlib.dates")
dates.AutoDateLocator = object
dates.DateFormatter = object
modules = {
"matplotlib": matplotlib,
"matplotlib.pyplot": pyplot,
"matplotlib.dates": dates,
}
with patch.dict(sys.modules, modules), patch.dict(os.environ, {"GITHUB_TOKEN": "test"}):
path = Path(__file__).with_name("star_history.py")
spec = importlib.util.spec_from_file_location("star_history", path)
module = importlib.util.module_from_spec(spec)
spec.loader.exec_module(module)
return module
star_history = load_star_history()
class StarHistoryTests(unittest.TestCase):
def http_error(self, code):
return HTTPError("https://api.github.com", code, "error", {}, None)
@patch.object(star_history.time, "sleep")
@patch.object(star_history.urllib.request, "urlopen")
def test_retries_server_error(self, urlopen, sleep):
response = object()
urlopen.side_effect = [self.http_error(500), response]
result = star_history.open_url(object())
self.assertIs(result, response)
self.assertEqual(urlopen.call_count, 2)
sleep.assert_called_once_with(1)
@patch.object(star_history.time, "sleep")
@patch.object(star_history.urllib.request, "urlopen")
def test_does_not_retry_non_transient_http_error(self, urlopen, sleep):
error = self.http_error(404)
urlopen.side_effect = error
with self.assertRaises(HTTPError):
star_history.open_url(object())
urlopen.assert_called_once()
sleep.assert_not_called()
@patch.object(star_history.time, "sleep")
@patch.object(star_history.urllib.request, "urlopen")
def test_stops_after_retry_limit(self, urlopen, sleep):
urlopen.side_effect = self.http_error(500)
with self.assertRaises(HTTPError):
star_history.open_url(object())
self.assertEqual(urlopen.call_count, star_history.REQUEST_ATTEMPTS)
self.assertEqual(sleep.call_count, star_history.REQUEST_ATTEMPTS - 1)
if __name__ == "__main__":
unittest.main()
+3 -3
View File
@@ -33,7 +33,7 @@ jobs:
# Initializes the CodeQL tools for scanning.
- name: Initialize CodeQL
uses: github/codeql-action/init@v4.39.1
uses: github/codeql-action/init@v4.39.3
# Override language selection by uncommenting this and choosing your languages
with:
languages: go
@@ -41,7 +41,7 @@ jobs:
# Autobuild attempts to build any compiled languages (C/C++, C#, or Java).
# If this step fails, then you should remove it and run the build manually (see below).
- name: Autobuild
uses: github/codeql-action/autobuild@v4.39.1
uses: github/codeql-action/autobuild@v4.39.3
# ℹ️ Command-line programs to run using the OS shell.
# 📚 See https://docs.github.com/en/actions/using-workflows/workflow-syntax-for-github-actions#jobsjob_idstepsrun
@@ -55,4 +55,4 @@ jobs:
# make release
- name: Perform CodeQL Analysis
uses: github/codeql-action/analyze@v4.39.1
uses: github/codeql-action/analyze@v4.39.3
+2 -2
View File
@@ -180,14 +180,14 @@ jobs:
- name: Login to Docker Hub
if: github.event_name != 'pull_request'
uses: docker/login-action@v4.7.0
uses: docker/login-action@v4.8.0
with:
username: ${{ secrets.DOCKER_USERNAME }}
password: ${{ secrets.DOCKER_PASSWORD }}
- name: Login to GHCR
if: github.event_name != 'pull_request'
uses: docker/login-action@v4.7.0
uses: docker/login-action@v4.8.0
with:
registry: ghcr.io
username: ${{ secrets.GHCR_USERNAME }}
@@ -141,7 +141,7 @@ jobs:
- name: Login to Docker Hub
if: github.event_name != 'pull_request'
uses: docker/login-action@v4.7.0
uses: docker/login-action@v4.8.0
with:
username: ${{ secrets.DOCKER_USERNAME }}
password: ${{ secrets.DOCKER_PASSWORD }}
+6 -6
View File
@@ -270,13 +270,13 @@ jobs:
buildkitd-config: /tmp/buildkitd.toml
- name: Login to Docker Hub
if: needs.setup.outputs.publish == 'true'
uses: docker/login-action@v4.7.0
uses: docker/login-action@v4.8.0
with:
username: ${{ secrets.DOCKER_USERNAME }}
password: ${{ secrets.DOCKER_PASSWORD }}
- name: Login to GHCR
if: needs.setup.outputs.publish == 'true'
uses: docker/login-action@v4.7.0
uses: docker/login-action@v4.8.0
with:
registry: ghcr.io
username: ${{ secrets.GHCR_USERNAME }}
@@ -324,7 +324,7 @@ jobs:
fi
- name: Login to GHCR
if: needs.setup.outputs.publish == 'true'
uses: docker/login-action@v4.7.0
uses: docker/login-action@v4.8.0
with:
registry: ghcr.io
username: ${{ secrets.GHCR_USERNAME }}
@@ -434,7 +434,7 @@ jobs:
output: trivy-results.sarif
exit-code: '0'
- name: Upload Trivy scan results to GitHub Security
uses: github/codeql-action/upload-sarif@v4.39.1
uses: github/codeql-action/upload-sarif@v4.39.3
if: always()
with:
sarif_file: trivy-results.sarif
@@ -501,12 +501,12 @@ jobs:
ghcr.io/chrislusf/seaweedfs
tags: type=raw,value=${{ github.event_name == 'workflow_dispatch' && github.event.inputs.image_tag || 'latest' }},suffix=${{ steps.config.outputs.tag_suffix }}
- name: Login to Docker Hub
uses: docker/login-action@v4.7.0
uses: docker/login-action@v4.8.0
with:
username: ${{ secrets.DOCKER_USERNAME }}
password: ${{ secrets.DOCKER_PASSWORD }}
- name: Login to GHCR
uses: docker/login-action@v4.7.0
uses: docker/login-action@v4.8.0
with:
registry: ghcr.io
username: ${{ secrets.GHCR_USERNAME }}
@@ -58,7 +58,7 @@ jobs:
-
name: Login to Docker Hub
if: github.event_name != 'pull_request'
uses: docker/login-action@v4.7.0
uses: docker/login-action@v4.8.0
with:
username: ${{ secrets.DOCKER_USERNAME }}
password: ${{ secrets.DOCKER_PASSWORD }}
@@ -262,7 +262,7 @@ jobs:
- name: Login to GHCR
if: github.event_name != 'workflow_dispatch' || github.event.inputs.variant == 'all' || github.event.inputs.variant == matrix.variant
uses: docker/login-action@v4.7.0
uses: docker/login-action@v4.8.0
with:
registry: ghcr.io
username: ${{ secrets.GHCR_USERNAME }}
@@ -352,7 +352,7 @@ jobs:
- name: Login to GHCR
if: github.event_name != 'workflow_dispatch' || github.event.inputs.variant == 'all' || github.event.inputs.variant == matrix.variant
uses: docker/login-action@v4.7.0
uses: docker/login-action@v4.8.0
with:
registry: ghcr.io
username: ${{ secrets.GHCR_USERNAME }}
@@ -374,7 +374,7 @@ jobs:
- name: Login to Docker Hub
if: github.event_name != 'workflow_dispatch' || github.event.inputs.variant == 'all' || github.event.inputs.variant == matrix.variant
uses: docker/login-action@v4.7.0
uses: docker/login-action@v4.8.0
with:
username: ${{ secrets.DOCKER_USERNAME }}
password: ${{ secrets.DOCKER_PASSWORD }}
@@ -487,7 +487,7 @@ jobs:
variant: large_disk
steps:
- name: Login to GHCR
uses: docker/login-action@v4.7.0
uses: docker/login-action@v4.8.0
with:
registry: ghcr.io
username: ${{ secrets.GHCR_USERNAME }}
@@ -515,7 +515,7 @@ jobs:
- name: Upload Trivy scan results to GitHub Security
if: always()
uses: github/codeql-action/upload-sarif@v4.39.1
uses: github/codeql-action/upload-sarif@v4.39.3
with:
sarif_file: trivy-results.sarif
category: trivy-${{ matrix.variant }}
@@ -539,13 +539,13 @@ jobs:
latest_tag: latest_large_disk
steps:
- name: Login to Docker Hub
uses: docker/login-action@v4.7.0
uses: docker/login-action@v4.8.0
with:
username: ${{ secrets.DOCKER_USERNAME }}
password: ${{ secrets.DOCKER_PASSWORD }}
- name: Login to GHCR
uses: docker/login-action@v4.7.0
uses: docker/login-action@v4.8.0
with:
registry: ghcr.io
username: ${{ secrets.GHCR_USERNAME }}
@@ -96,7 +96,7 @@ jobs:
uses: docker/setup-buildx-action@4d04d5d9486b7bd6fa91e7baf45bbb4f8b9deedd # v4.0.0
- name: Login to Docker Hub
uses: docker/login-action@81c6f93d4bc991e510b342a00b419b3bb4efc5c6 # v4.7.0
uses: docker/login-action@8aa5831fb8d48d8f76594c7ddde61ae58019f550 # v4.8.0
with:
username: ${{ secrets.DOCKER_USERNAME }}
password: ${{ secrets.DOCKER_PASSWORD }}
+3
View File
@@ -61,6 +61,9 @@ jobs:
restore-keys: |
${{ runner.os }}-buildx-e2e-
- name: Test APT mirror fallback
run: python3 test_apt_install.py
- name: Install dependencies
run: |
sudo rm -f /etc/apt/sources.list.d/azure-cli.list /etc/apt/sources.list.d/microsoft-prod.list
+69 -1
View File
@@ -37,6 +37,13 @@ jobs:
with:
version: v3.18.4
- uses: actions/setup-go@v6
with:
go-version-file: go.mod
- name: Test vector chart resources
run: go test ./k8s/charts/seaweedfs/tests -count=1
- uses: actions/setup-python@v7
with:
python-version: '3.10'
@@ -1321,6 +1328,67 @@ jobs:
echo "$BOOL_FALSE" | grep -q -- '-status Suspended' || { echo "FAIL: bool false versioning did not Suspend the bucket"; exit 1; }
echo "Bucket versioning: YAML bool false suspends consistently with string \"false\""
echo "=== Testing bucket ttl, replication and disk in one fs.configure ==="
FS_CONFIGURE=$(helm template test $CHART_DIR \
--set s3.enabled=true \
--set s3.createBuckets[0].name=cfgbucket \
--set s3.createBuckets[0].ttl=7d \
--set-string s3.createBuckets[0].replication=001 \
--set s3.createBuckets[0].disk=ssd | grep 'fs.configure -locationPrefix=/buckets/cfgbucket/' || true)
echo "$FS_CONFIGURE" | grep -q -- '-ttl=7d -replication=001 -disk=ssd -apply' || { echo "FAIL: bucket fs.configure missing ttl/replication/disk: $FS_CONFIGURE"; exit 1; }
echo "Bucket fs.configure: ttl, replication and disk render together"
FS_CONFIGURE=$(helm template test $CHART_DIR \
--set s3.enabled=true \
--set s3.createBuckets[0].name=replbucket \
--set-string s3.createBuckets[0].replication=002 | grep 'fs.configure -locationPrefix=/buckets/replbucket/' || true)
echo "$FS_CONFIGURE" | grep -q -- '/buckets/replbucket/ -replication=002 -apply' || { echo "FAIL: replication-only bucket did not render fs.configure: $FS_CONFIGURE"; exit 1; }
echo "Bucket fs.configure: replication without ttl"
FS_CONFIGURE=$(helm template test $CHART_DIR \
--set s3.enabled=true \
--set s3.createBuckets[0].name=diskbucket \
--set s3.createBuckets[0].disk=ssd | grep 'fs.configure -locationPrefix=/buckets/diskbucket/' || true)
echo "$FS_CONFIGURE" | grep -q -- '/buckets/diskbucket/ -disk=ssd -apply' || { echo "FAIL: disk-only bucket did not render fs.configure: $FS_CONFIGURE"; exit 1; }
echo "Bucket fs.configure: disk without ttl"
if helm template test $CHART_DIR \
--set s3.enabled=true \
--set s3.createBuckets[0].name=numbucket \
--set s3.createBuckets[0].replication=0 >/dev/null 2>&1; then
echo "FAIL: a numeric replication should fail to render"; exit 1
fi
echo "Bucket fs.configure: numeric replication is rejected"
if helm template test $CHART_DIR \
--set s3.enabled=true \
--set s3.createBuckets[0].name=badbucket \
--set-string s3.createBuckets[0].replication=00O >/dev/null 2>&1; then
echo "FAIL: a non-digit replication should fail to render"; exit 1
fi
echo "Bucket fs.configure: non-digit replication is rejected"
if helm template test $CHART_DIR \
--set s3.enabled=true \
--set s3.createBuckets[0].name=bigbucket \
--set-string s3.createBuckets[0].replication=256 >/dev/null 2>&1; then
echo "FAIL: a replication above 255 should fail to render"; exit 1
fi
FS_CONFIGURE=$(helm template test $CHART_DIR \
--set s3.enabled=true \
--set s3.createBuckets[0].name=maxbucket \
--set-string s3.createBuckets[0].replication=255 | grep 'fs.configure -locationPrefix=/buckets/maxbucket/' || true)
echo "$FS_CONFIGURE" | grep -q -- '/buckets/maxbucket/ -replication=255 -apply' || { echo "FAIL: replication 255 should render: $FS_CONFIGURE"; exit 1; }
echo "Bucket fs.configure: replication is limited to 000-255"
FS_CONFIGURE=$(helm template test $CHART_DIR \
--set s3.enabled=true \
--set s3.createBuckets[0].name=nullbucket \
--set s3.createBuckets[0].ttl=7d \
--set s3.createBuckets[0].replication=null | grep 'fs.configure -locationPrefix=/buckets/nullbucket/' || true)
echo "$FS_CONFIGURE" | grep -q -- '/buckets/nullbucket/ -ttl=7d -apply' || { echo "FAIL: null replication should be treated as unset: $FS_CONFIGURE"; exit 1; }
echo "Bucket fs.configure: null replication is unset"
echo ""
echo "=== Testing hook Job labels ==="
# The hook Jobs were the only pods in the chart without the standard
@@ -2131,7 +2199,7 @@ jobs:
echo "IMAGE_TAG=$TAG" >> $GITHUB_ENV
- name: Create kind cluster
uses: helm/kind-action@v1.15.1
uses: helm/kind-action@v1.15.2
- name: Verify volume resize hook security contexts
run: |
+135
View File
@@ -0,0 +1,135 @@
name: "rust: vector desktop platforms"
on:
pull_request:
paths:
- 'seaweed-vector/**'
- 'seaweed-worker/**'
- 'seaweed-common/**'
- 'weed/pb/**'
- '.github/workflows/rust-vector-platforms.yml'
- 'install.sh'
- 'test/installer/**'
push:
tags: ['*']
workflow_dispatch:
permissions:
contents: read
concurrency:
group: vector-platforms-${{ github.event.pull_request.number || github.ref }}
cancel-in-progress: ${{ github.event_name == 'pull_request' }}
jobs:
vector:
permissions:
contents: read
strategy:
fail-fast: false
matrix:
include:
- runner: macos-15-intel
suffix: darwin_amd64
protoc: macos-x86_64
- runner: macos-latest
suffix: darwin_arm64
protoc: macos-aarch_64
- runner: windows-latest
suffix: windows_amd64
protoc: win32
runs-on: ${{ matrix.runner }}
timeout-minutes: 120
env:
CARGO_BUILD_JOBS: '2'
CARGO_PROFILE_TEST_DEBUG: '0'
SEAWEEDFS_COMMIT: ${{ github.sha }}
defaults:
run:
shell: bash
steps:
- uses: actions/checkout@v7
with:
persist-credentials: false
- uses: dtolnay/rust-toolchain@stable
- uses: actions/setup-python@v6
with:
python-version: '3.13'
- uses: actions/cache/restore@v6
id: cargo-cache
if: ${{ !startsWith(github.ref, 'refs/tags/') }}
with:
path: |
~/.cargo/registry
~/.cargo/git
seaweed-vector/target
key: vector-${{ matrix.suffix }}-${{ hashFiles('seaweed-vector/Cargo.lock') }}
restore-keys: vector-${{ matrix.suffix }}-
- name: Fetch dependencies
run: cargo fetch --locked --manifest-path seaweed-vector/Cargo.toml
- name: Select vendored protoc
env:
PROTOC_PACKAGE: protoc-bin-vendored-${{ matrix.protoc }}
run: |
python - <<'PY'
import os, pathlib, tomllib
lock=tomllib.loads(pathlib.Path('seaweed-vector/Cargo.lock').read_text())
name=os.environ['PROTOC_PACKAGE']
version=next(p['version'] for p in lock['package'] if p['name']==name)
cargo=pathlib.Path(os.environ.get('CARGO_HOME',pathlib.Path.home()/'.cargo'))
executable='protoc.exe' if os.name=='nt' else 'protoc'
path=next(cargo.glob(f'registry/src/*/{name}-{version}/bin/{executable}'))
with open(os.environ['GITHUB_ENV'],'a') as out: out.write(f'PROTOC={path}\n')
PY
- name: Build vector engine
id: cargo-build
run: cargo build --locked --release --manifest-path seaweed-vector/Cargo.toml -p weed-vector-server
- name: Test vector crates
run: cargo test --locked --release --manifest-path seaweed-vector/Cargo.toml --workspace
- name: Test installer
run: |
if [ "$RUNNER_OS" = "Windows" ]; then
BASH_BINARY="$(cygpath -w "$BASH")"
export BASH_BINARY
fi
python -m unittest discover -s test/installer
- name: Save compiled dependencies
if: ${{ always() && !startsWith(github.ref, 'refs/tags/') && steps.cargo-build.outcome == 'success' && steps.cargo-cache.outputs.cache-hit != 'true' }}
uses: actions/cache/save@v6
with:
path: |
~/.cargo/registry
~/.cargo/git
seaweed-vector/target
key: vector-${{ matrix.suffix }}-${{ hashFiles('seaweed-vector/Cargo.lock') }}
- name: Package engine
env:
ASSET_SUFFIX: ${{ matrix.suffix }}
run: |
python - <<'PY'
import hashlib, os, pathlib, tarfile, zipfile
suffix=os.environ['ASSET_SUFFIX']
binary='weed-vector.exe' if os.name=='nt' else 'weed-vector'
source=pathlib.Path('seaweed-vector/target/release')/binary
asset=f'weed-vector_{suffix}'+('.zip' if os.name=='nt' else '.tar.gz')
if os.name=='nt':
with zipfile.ZipFile(asset,'w',zipfile.ZIP_DEFLATED) as archive: archive.write(source,binary)
else:
with tarfile.open(asset,'w:gz') as archive: archive.add(source,arcname=binary)
digest=hashlib.md5(pathlib.Path(asset).read_bytes()).hexdigest()
pathlib.Path(asset+'.md5').write_text(f'{digest} {asset}\n')
PY
- uses: actions/upload-artifact@v7
with:
name: vector-${{ matrix.suffix }}
path: weed-vector_${{ matrix.suffix }}.*
release:
if: startsWith(github.ref, 'refs/tags/')
needs: vector
runs-on: ubuntu-latest
permissions:
contents: write
steps:
- uses: actions/download-artifact@v8
with:
pattern: vector-*
merge-multiple: true
- uses: softprops/action-gh-release@v3
with:
files: weed-vector_*
+30 -4
View File
@@ -76,24 +76,37 @@ jobs:
vectors-e2e:
name: S3 Vectors Integration (${{ matrix.volume }} volume)
name: S3 Vectors Integration (${{ matrix.volume }} volume${{ matrix.hardened && ', hardened' || '' }}${{ matrix.worker && ', worker' || '' }})
runs-on: ubuntu-24.04
timeout-minutes: 45
strategy:
fail-fast: false
matrix:
volume: [go, rust]
# Hardened: volume servers behind mutual TLS, every gRPC hop (master,
# volume, filer, gateway, engine) under mutual TLS, read JWTs
# required, and every chunk encrypted by the gateway.
hardened: ["", "1"]
worker: ["", "1"]
# Worker: the admin server and the Lance maintenance worker join the
# cluster, for the tests that build vector indexes through a job.
env:
CARGO_PROFILE_DEV_DEBUG: "0"
CARGO_BUILD_JOBS: "2"
CARGO_TARGET_DIR: ${{ github.workspace }}/seaweed-vector/target
VOLUME_SERVER_IMPL: ${{ matrix.volume }}
VECTOR_QUERY_MODE: single
VECTOR_FILER_GROUP: vectors-ci
VECTOR_TLS: ${{ matrix.hardened }}
VECTOR_READ_JWT: ${{ matrix.hardened }}
VECTOR_ENCRYPT_VOLUMES: ${{ matrix.hardened }}
VECTOR_WORKER: ${{ matrix.worker }}
steps:
- uses: actions/checkout@v7
with:
persist-credentials: false
- uses: actions/setup-go@v5
- uses: actions/setup-go@v7
with:
go-version-file: go.mod
cache: ${{ runner.environment == 'github-hosted' }}
@@ -116,6 +129,18 @@ jobs:
run: |
go test ./weed/s3api/s3vectors -count=1
go test ./weed/command -run 'Vector|Vectors' -count=1
# The worker workspace has no protoc of its own for lance's build
# scripts; use the vendored one its lock names, as its own workflow does.
- name: Use the vendored protoc for the worker
if: matrix.worker == '1'
run: |
cd seaweed-worker
cargo fetch --locked
version=$(awk '/^name = "protoc-bin-vendored-linux-x86_64"$/{found=1; next} found && /^version = /{gsub(/"/,"",$3); print $3; exit}' Cargo.lock)
test -n "$version" || { echo "protoc-bin-vendored-linux-x86_64 is not in Cargo.lock" >&2; exit 1; }
protoc=$(find ~/.cargo/registry/src -path "*protoc-bin-vendored-linux-x86_64-$version/bin/protoc" | head -1)
test -x "$protoc" || { echo "no vendored protoc $version in the registry" >&2; exit 1; }
echo "PROTOC=$protoc" >> "$GITHUB_ENV"
- name: Build integration binaries
working-directory: test/s3vectors
# dev profile: correctness suite, no release codegen needed. weed-volume
@@ -124,7 +149,8 @@ jobs:
set -e
make ./weed ./weed-vector CARGO_PROFILE=dev
if [ "${{ matrix.volume }}" = "rust" ]; then make ./weed-volume CARGO_PROFILE=dev; fi
- uses: actions/setup-python@v5
if [ "${{ matrix.worker }}" = "1" ]; then make ./weed-vector-cli ./weed-worker CARGO_PROFILE=dev; fi
- uses: actions/setup-python@v7
with:
python-version: "3.12"
- name: Install test dependencies
@@ -147,7 +173,7 @@ jobs:
if: always()
uses: actions/upload-artifact@v7
with:
name: vectors-e2e-diagnostics-${{ matrix.volume }}
name: vectors-e2e-diagnostics-${{ matrix.volume }}${{ matrix.hardened && '-hardened' || '' }}${{ matrix.worker && '-worker' || '' }}
path: |
${{ runner.temp }}/vector-tests
${{ runner.temp }}/vector-results.xml
+2 -1
View File
@@ -147,6 +147,7 @@ telemetry/server/telemetry-server
__pycache__/
/test/s3vectors/weed
/test/s3vectors/weed-vector
/test/s3vectors/weed-vector-worker
/test/s3vectors/weed-vector-cli
/test/s3vectors/weed-worker
/test/s3vectors/weed-volume
.pytest_cache/
+41 -3
View File
@@ -17,7 +17,9 @@ SeaweedFS is a simple and highly scalable distributed file system. There are two
1. to store billions of files!
2. to serve the files fast!
One `weed` binary serves an S3 object store, a POSIX file system, and a lakehouse with S3 Tables, all over the same data. Each blob is one disk read away, capacity grows by starting another volume server, and cloud storage can be cached or tiered transparently. Both read and write operations have O(1) complexity and can run at the full speed supported by the underlying hardware.
One SeaweedFS cluster serves an S3 object store, a POSIX file system, a lakehouse with S3 Tables and a built-in Iceberg REST Catalog, and [vector search with S3 Vector Buckets](#vector-search-with-s3-vector-buckets), all on shared storage. Each blob is one disk read away, capacity grows by starting another volume server, and cloud storage can be cached or tiered transparently. Blob reads and writes have O(1) complexity and can run at the full speed supported by the underlying hardware.
![SeaweedFS feature overview: S3 objects, POSIX files, table buckets with an Iceberg REST Catalog and Lance namespace, and S3 Vector Buckets share one cluster of masters, filers, and volume servers, with cloud caching, tiering, and cross-cluster replication.](note/seaweedfs-features.svg)
- [Download Binaries for different platforms](https://github.com/seaweedfs/seaweedfs/releases/latest)
- [Wiki Documentation](https://github.com/seaweedfs/seaweedfs/wiki)
@@ -40,6 +42,7 @@ Table of Contents
* [Scalable](#scalable)
* [The most complete S3 API](#the-most-complete-s3-api)
* [A data warehouse with S3 Tables](#a-data-warehouse-with-s3-tables)
* [Vector search with S3 Vector Buckets](#vector-search-with-s3-vector-buckets)
* [A fast cache for cloud storage](#a-fast-cache-for-cloud-storage)
* [Active-active replication and more](#active-active-replication-and-more)
* [Architecture](#architecture)
@@ -204,12 +207,13 @@ On one laptop, [`weed benchmark`][Benchmarks] writes 1KB files at 15,700 per sec
## The most complete S3 API ##
The S3 gateway implements the object, bucket, S3 Tables, IAM, and STS APIs on one endpoint, so the AWS SDKs and CLI, rclone, restic, Spark, and Trino work unchanged.
The S3 gateway implements the object, bucket, S3 Tables, S3 Vectors, IAM, and STS APIs on one endpoint, so the AWS SDKs and CLI, rclone, restic, Spark, and Trino work unchanged. S3 Vectors uses the `weed-vector` service described [below](#vector-search-with-s3-vector-buckets).
| API | Operations |
| --- | --- |
| S3 bucket and object | 73 |
| S3 Tables | 36 |
| S3 Vectors | 19 |
| IAM | 39 |
| STS | 5 |
@@ -223,7 +227,7 @@ The full operation list is in [Amazon S3 API][AmazonS3API], and [Supported APIs
## A data warehouse with S3 Tables ##
SeaweedFS is a lakehouse in one system. [S3 Table Buckets][S3TableBucket] hold Apache Iceberg tables by default, or [Lance][LanceCatalog] tables for vectors and multimodal data, and the built-in [Iceberg REST Catalog][IcebergCatalog] and Lance namespace serve them directly. There is no Hive Metastore, Glue, or separate catalog service to deploy, secure, and back up.
SeaweedFS is a lakehouse in one system. [S3 Table Buckets][S3TableBucket] hold Apache Iceberg tables by default, or [Lance][LanceCatalog] tables for vectors and multimodal data, and the built-in [Iceberg REST Catalog][IcebergCatalog] and Lance namespace serve them directly. The S3 Tables API and Iceberg REST Catalog share the same namespaces, table metadata, and permissions. There is no Hive Metastore, Glue, or separate catalog service to deploy, secure, and back up.
* Query engines operate on the same tables at the same time: [Spark][SparkIceberg], [Trino][TrinoIceberg], [Dremio][DremioIceberg], [DuckDB][DuckDBIceberg], [Apache Doris][DorisIceberg], [RisingWave][RisingWaveIceberg], ClickHouse, and [LanceDB][LanceDB]. Catalog commits are atomic compare-and-swap, so concurrent writers are safe. [Lakekeeper][Lakekeeper] can front the same storage with STS-vended credentials. [OLake Go][OLake] is a high-performance, open-source EL (Extract–Load) platform that bridges operational databases and open lakehouse storage - writing Apache Iceberg tables or Parquet files on Amazon S3.
* [Automated table maintenance][IcebergMaintenance]: compaction, snapshot expiration, orphan file removal, and manifest rewriting, configured per bucket or table through the S3 Tables maintenance APIs, and the same for [Lance][LanceMaintenance].
@@ -232,6 +236,40 @@ SeaweedFS is a lakehouse in one system. [S3 Table Buckets][S3TableBucket] hold A
`S3_TABLE_BUCKET=warehouse ./weed mini -dir=./data` brings the whole stack up on a laptop.
## Vector search with S3 Vector Buckets ##
Keep embeddings alongside your objects and tables in the same SeaweedFS cluster. S3 Vector Buckets expose the AWS S3 Vectors API for semantic search, retrieval-augmented generation (RAG), and recommendations, using Lance tables on SeaweedFS storage.
* Create vector buckets and indexes; insert, retrieve, list, and delete vectors with the AWS SDKs or CLI pointed at your S3 endpoint.
* Find the nearest vectors with cosine or Euclidean distance, filter by metadata, and return distances and metadata with the results.
* Control access with `s3vectors:*` IAM actions and vector bucket policies; tag buckets and indexes for organization and access control.
Install the separate vector engine; the default installer installs only `weed`:
```bash
curl -fsSL https://raw.githubusercontent.com/seaweedfs/seaweedfs/master/install.sh | bash -s -- --component vector
```
Release 4.53 provides vector binaries for Linux amd64 and arm64. If your release has no binary for your platform, build from [source](seaweed-vector). From a directory where you want a new checkout:
```bash
git clone https://github.com/seaweedfs/seaweedfs.git
cd seaweedfs
(cd seaweed-vector && cargo build --release -p weed-vector-server)
```
If you already have a checkout, start at its root and skip the clone and `cd seaweedfs` steps.
For Windows source builds, set `PROTOC` to a native `protoc.exe` before running Cargo; the [desktop build workflow](.github/workflows/rust-vector-platforms.yml) shows how to select the vendored executable.
Run the installed `weed-vector` binary alongside the local cluster started with `weed mini` above (or use `seaweed-vector/target/release/weed-vector` after building from source):
```bash
weed-vector --masters=localhost:9333
```
It registers with the master and the S3 gateway discovers it automatically. An AWS S3 Vectors client uses the same S3 endpoint at http://localhost:8333. The vector engine stores its data through the cluster's filers and volume servers, so embeddings share the storage layer's replication and erasure coding. See the [AWS SDK examples](test/s3vectors/s3vectors_test.py) for bucket, index, and query operations.
## A fast cache for cloud storage ##
[Cloud Drive][CloudDrive] mounts a bucket from S3, Google Cloud Storage, Azure, Backblaze B2, Wasabi, Storj, or any S3-compatible store into SeaweedFS and serves it at local speed:
+4
View File
@@ -271,6 +271,10 @@ curl "http://localhost:9333/vol/vacuum?garbageThreshold=0.4"
Vacuuming makes a volume read-only, copies live needles to a new volume, and
swaps it in.
The master runs one vacuum at a time. A request that arrives while the
automatic vacuum or another request is running does nothing and gets
`409 Conflict` with `{"error":"another vacuum is running, try again later"}`.
### Delete a collection
```bash
+8 -10
View File
@@ -3,17 +3,15 @@
# unreachable. See docker/Dockerfile.e2e.
set -e
# Every rewrite starts from the pristine list, so a mirror that just failed does
# not become the pattern the next rewrite has to match.
[ -f /etc/apt/sources.list.orig ] || cp /etc/apt/sources.list /etc/apt/sources.list.orig
for mirror in azure.archive.ubuntu.com archive.ubuntu.com; do
# Any archive host, so this works whether the pristine list came from the
# base image (archive.ubuntu.com) or a CI runner (azure.archive.ubuntu.com).
sed "s|http://[a-z0-9.]*archive\.ubuntu\.com/ubuntu|http://$mirror/ubuntu|g; s|http://security\.ubuntu\.com/ubuntu|http://$mirror/ubuntu|g" \
/etc/apt/sources.list.orig > /etc/apt/sources.list
if apt-get -o Acquire::Retries=3 -o Acquire::http::Timeout=15 update && \
DEBIAN_FRONTEND=noninteractive apt-get -o Acquire::Retries=3 -o Acquire::http::Timeout=15 install -y \
for apt_source in /etc/apt/sources.list /etc/apt/sources.list.d/*.list /etc/apt/sources.list.d/*.sources; do
[ -f "$apt_source" ] || continue
[ -f "$apt_source.orig" ] || cp "$apt_source" "$apt_source.orig"
sed -E "s|(https?://)[a-z0-9.]*archive\.ubuntu\.com/ubuntu|\1$mirror/ubuntu|g; s|(https?://)security\.ubuntu\.com/ubuntu|\1$mirror/ubuntu|g" \
"$apt_source.orig" > "$apt_source"
done
if timeout --kill-after=5s 120s apt-get -o Acquire::Retries=1 -o Acquire::http::Timeout=15 -o Acquire::https::Timeout=15 -o APT::Update::Error-Mode=any update && \
DEBIAN_FRONTEND=noninteractive timeout --kill-after=5s 180s apt-get -o Acquire::Retries=1 -o Acquire::http::Timeout=15 -o Acquire::https::Timeout=15 install -y \
--no-install-recommends --no-install-suggests "$@"; then
apt-get clean
rm -rf /var/lib/apt/lists/*
+8 -8
View File
@@ -64,7 +64,7 @@ case "$1" in
'master')
ARGS="-mdir=/data -volumeSizeLimitMB=1024"
shift
exec /usr/bin/weed -logtostderr=true master $ARGS $@
exec /usr/bin/weed -logtostderr=true master $ARGS "$@"
;;
'volume')
@@ -73,7 +73,7 @@ case "$1" in
ARGS="-dir=/data"
fi
shift
exec /usr/bin/weed -logtostderr=true volume $ARGS $@
exec /usr/bin/weed -logtostderr=true volume $ARGS "$@"
;;
'volume-rust')
@@ -87,7 +87,7 @@ case "$1" in
echo "Use 'volume' for the Go volume server instead." >&2
exit 1
fi
exec /usr/bin/weed-volume $ARGS $@
exec /usr/bin/weed-volume $ARGS "$@"
;;
'worker-rust')
@@ -119,7 +119,7 @@ case "$1" in
ARGS="-dir=/data -master.volumeSizeLimitMB=1024"
fi
shift
exec /usr/bin/weed -logtostderr=true server $ARGS $@
exec /usr/bin/weed -logtostderr=true server $ARGS "$@"
;;
'mini')
@@ -128,19 +128,19 @@ case "$1" in
ARGS=""
fi
shift
exec /usr/bin/weed -logtostderr=true mini $ARGS $@
exec /usr/bin/weed -logtostderr=true mini $ARGS "$@"
;;
'filer')
ARGS=""
shift
exec /usr/bin/weed -logtostderr=true filer $ARGS $@
exec /usr/bin/weed -logtostderr=true filer $ARGS "$@"
;;
's3')
ARGS="-domainName=$S3_DOMAIN_NAME -key.file=$S3_KEY_FILE -cert.file=$S3_CERT_FILE"
shift
exec /usr/bin/weed -logtostderr=true s3 $ARGS $@
exec /usr/bin/weed -logtostderr=true s3 $ARGS "$@"
;;
'shell')
@@ -150,6 +150,6 @@ case "$1" in
;;
*)
exec /usr/bin/weed $@
exec /usr/bin/weed "$@"
;;
esac
+39
View File
@@ -0,0 +1,39 @@
# Vector buckets locally
Run on Linux containers (amd64 or arm64):
```sh
docker compose up -d
export AWS_ACCESS_KEY_ID=vector-demo
export AWS_SECRET_ACCESS_KEY=vector-demo-secret
export AWS_DEFAULT_REGION=us-east-1
aws --endpoint-url http://localhost:8333 s3vectors create-index \
--vector-bucket-name embeddings --index-name images \
--data-type float32 --dimension 3 --distance-metric cosine
aws --endpoint-url http://localhost:8333 s3vectors put-vectors \
--vector-bucket-name embeddings --index-name images \
--vectors '[{"key":"first","data":{"float32":[1,0,0]}}]'
aws --endpoint-url http://localhost:8333 s3vectors query-vectors \
--vector-bucket-name embeddings --index-name images \
--query-vector '{"float32":[1,0,0]}' --top-k 1
```
The engine registers with the master. Wait for that registration before creating
an index. The Lance worker builds and refreshes ANN indexes as data grows;
queries can scan data before an ANN index is built. The demo's admin UI is at
http://localhost:12535 (user `admin`, password `vector-demo`). Demo credentials
and plaintext cluster connections are for this local example. Production
settings are in the Helm/operator TLS and authentication documentation.
`docker compose down` preserves the named data volume. Add `-v` to discard it.
Override `SEAWEEDFS_IMAGE` to test a newer image. With metrics-enabled engines,
append `--metrics-port=9329 --metrics-ip=0.0.0.0` to the vector command and scrape
`vector:9329/metrics` inside the Compose network.
A native development setup can use `weed mini -vector -vectorBucket=embeddings`
with `weed-vector` beside `weed` or on PATH. `-vector.binary` selects an explicit
binary, and `-vector.metricsPort=9329` enables HTTP readiness and metrics on a
compatible engine. The child inherits the cluster's configuration directory and
stops with mini. Run `weed-worker` separately for ANN maintenance. `weed shell`
provides `s3vectors -operation <Operation> -file request.json` for the same API
operations using trusted cluster credentials.
+34
View File
@@ -0,0 +1,34 @@
services:
seaweedfs:
image: ${SEAWEEDFS_IMAGE:-chrislusf/seaweedfs:4.53}
command: mini -dir=/data -ip=seaweedfs -ip.bind=0.0.0.0 -admin.password=vector-demo -admin.worker.ip=0.0.0.0 -tableBucket=embeddings:LANCE
environment: &environment
WEED_JWT_FILER_SIGNING_KEY: vector-demo-cluster-key
AWS_ACCESS_KEY_ID: vector-demo
AWS_SECRET_ACCESS_KEY: vector-demo-secret
ports:
- "127.0.0.1:8333:8333"
- "127.0.0.1:12535:12535"
volumes:
- data:/data
healthcheck:
test: [CMD-SHELL, "wget -q -O /dev/null http://127.0.0.1:9333/cluster/status"]
interval: 5s
timeout: 3s
retries: 30
vector:
image: ${SEAWEEDFS_IMAGE:-chrislusf/seaweedfs:4.53}
command: vector --listen=vector:8090 --masters=seaweedfs:9333
environment: *environment
depends_on:
seaweedfs:
condition: service_healthy
worker:
image: ${SEAWEEDFS_IMAGE:-chrislusf/seaweedfs:4.53}
command: worker-rust --admin=seaweedfs:12535 --masters=seaweedfs:9333 --namespace=http://seaweedfs:9101
environment: *environment
depends_on:
seaweedfs:
condition: service_healthy
volumes:
data:
+110
View File
@@ -0,0 +1,110 @@
import json
import os
from pathlib import Path
import subprocess
import tempfile
import unittest
class AptInstallTests(unittest.TestCase):
def run_helper(self, sources, fail_action="update", timeout_azure=False):
with tempfile.TemporaryDirectory() as directory:
root = Path(directory)
apt = root / "apt"
(apt / "sources.list.d").mkdir(parents=True)
(apt / "sources.list").write_text("")
for name, content in sources.items():
(apt / name).write_text(content)
helper = root / "apt-install"
helper.write_text(Path(__file__).with_name("apt-install").read_text()
.replace("/etc/apt", str(apt))
.replace("/var/lib/apt", str(root / "state")))
commands = root / "bin"
commands.mkdir()
fake_apt = commands / "apt-get"
fake_apt.write_text(r"""#!/usr/bin/env python3
import json, os, pathlib, sys
root = pathlib.Path(os.environ["VECTOR_APT_TEST_ROOT"])
apt = root / "apt"
sources = [apt / "sources.list", *apt.glob("sources.list.d/*.list"), *apt.glob("sources.list.d/*.sources")]
content = "".join(p.read_text() for p in sources)
action = next(a for a in sys.argv[1:] if a in ("update", "install", "clean"))
with (root / "calls").open("a") as output:
output.write(json.dumps({"action": action, "sources": content, "args": sys.argv[1:]}) + "\n")
if "azure.archive.ubuntu.com" in content and action == "update" and os.environ["VECTOR_APT_FAIL_ACTION"] == "partial":
sys.exit(100 if "APT::Update::Error-Mode=any" in sys.argv else 0)
if "azure.archive.ubuntu.com" in content and action == os.environ["VECTOR_APT_FAIL_ACTION"]:
sys.exit(100)
""")
fake_apt.chmod(0o755)
fake_timeout = commands / "timeout"
fake_timeout.write_text(r"""#!/usr/bin/env python3
import json, os, pathlib, sys
root = pathlib.Path(os.environ["VECTOR_APT_TEST_ROOT"])
apt = root / "apt"
sources = [apt / "sources.list", *apt.glob("sources.list.d/*.list"), *apt.glob("sources.list.d/*.sources")]
content = "".join(p.read_text() for p in sources)
with (root / "calls").open("a") as output:
output.write(json.dumps({"action": "timeout", "args": sys.argv[1:]}) + "\n")
if os.environ["VECTOR_APT_TIMEOUT_AZURE"] and "azure.archive.ubuntu.com" in content:
sys.exit(124)
os.execvp(sys.argv[3], sys.argv[3:])
""")
fake_timeout.chmod(0o755)
result = subprocess.run(["/bin/sh", str(helper), "fuse"],
env={**os.environ, "PATH": str(commands) + os.pathsep + os.environ["PATH"],
"VECTOR_APT_TEST_ROOT": str(root), "VECTOR_APT_FAIL_ACTION": fail_action,
"VECTOR_APT_TIMEOUT_AZURE": "1" if timeout_azure else ""},
text=True, capture_output=True, timeout=5)
calls = [json.loads(line) for line in (root / "calls").read_text().splitlines()] if (root / "calls").exists() else []
return result, calls
def test_fallback_rewrites_all_source_formats(self):
for protocol in ("http", "https"):
for layout in ("legacy", "deb822", "mixed"):
with self.subTest(protocol=protocol, layout=layout):
sources = {}
if layout in ("legacy", "mixed"):
sources["sources.list"] = f"deb {protocol}://azure.archive.ubuntu.com/ubuntu noble main\n"
if layout in ("deb822", "mixed"):
sources["sources.list.d/ubuntu.sources"] = f"Types: deb\nURIs: {protocol}://azure.archive.ubuntu.com/ubuntu\nSuites: noble\nComponents: main\n"
result, calls = self.run_helper(sources)
self.assertEqual(result.returncode, 0, result.stderr)
installs = [call for call in calls if call["action"] == "install"]
self.assertEqual(len(installs), 1)
self.assertNotIn("azure.archive.ubuntu.com", installs[0]["sources"])
self.assertIn(protocol + "://archive.ubuntu.com", installs[0]["sources"])
def test_install_failure_also_tries_the_next_mirror(self):
result, calls = self.run_helper({"sources.list.d/ubuntu.sources":
"Types: deb\nURIs: http://security.ubuntu.com/ubuntu\nSuites: noble-security\nComponents: main\n"},
fail_action="install")
self.assertEqual(result.returncode, 0, result.stderr)
installs = [call for call in calls if call["action"] == "install"]
self.assertEqual(len(installs), 2)
self.assertIn("azure.archive.ubuntu.com", installs[0]["sources"])
self.assertNotIn("azure.archive.ubuntu.com", installs[1]["sources"])
def test_partial_update_does_not_install_stale_indexes(self):
result, calls = self.run_helper({"sources.list": "deb http://azure.archive.ubuntu.com/ubuntu noble main\n"},
fail_action="partial")
self.assertEqual(result.returncode, 0, result.stderr)
installs = [call for call in calls if call["action"] == "install"]
self.assertEqual(len(installs), 1)
self.assertNotIn("azure.archive.ubuntu.com", installs[0]["sources"])
def test_timeout_advances_to_the_next_mirror(self):
result, calls = self.run_helper({"sources.list": "deb http://azure.archive.ubuntu.com/ubuntu noble main\n"},
fail_action="none", timeout_azure=True)
self.assertEqual(result.returncode, 0, result.stderr)
timeouts = [call for call in calls if call["action"] == "timeout"]
self.assertEqual(len(timeouts), 3)
for call in timeouts:
self.assertEqual(call["args"][0], "--kill-after=5s")
self.assertLessEqual(int(call["args"][1].removesuffix("s")), 180)
installs = [call for call in calls if call["action"] == "install"]
self.assertNotIn("azure.archive.ubuntu.com", installs[0]["sources"])
if __name__ == "__main__":
unittest.main()
+47 -47
View File
@@ -32,7 +32,7 @@ require (
github.com/google/btree v1.1.3
github.com/google/uuid v1.6.0
github.com/google/wire v0.7.0 // indirect
github.com/googleapis/gax-go/v2 v2.24.1 // indirect
github.com/googleapis/gax-go/v2 v2.27.0 // indirect
github.com/gorilla/mux v1.8.1
github.com/hashicorp/errwrap v1.1.0 // indirect
github.com/hashicorp/go-multierror v1.1.1 // indirect
@@ -53,7 +53,7 @@ require (
github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd // indirect
github.com/modern-go/reflect2 v1.0.2 // indirect
github.com/olivere/elastic/v7 v7.0.32
github.com/peterh/liner v1.2.2
github.com/peterh/liner v1.2.3
github.com/pkg/errors v0.9.1 // indirect
github.com/pmezard/go-difflib v1.0.1-0.20181226105442-5d4384ee4fb2 // indirect
github.com/posener/complete v1.2.3
@@ -61,11 +61,11 @@ require (
github.com/prometheus/client_golang v1.24.1
github.com/prometheus/client_model v0.6.3
github.com/prometheus/common v0.71.0 // indirect
github.com/prometheus/procfs v0.22.0
github.com/prometheus/procfs v0.23.0
github.com/rcrowley/go-metrics v0.0.0-20201227073835-cf1acfcdf475 // indirect
github.com/remyoudompheng/bigfft v0.0.0-20230129092748-24d4a6f8daec // indirect
github.com/seaweedfs/goexif v1.0.3
github.com/seaweedfs/raft v1.2.3
github.com/seaweedfs/raft v1.2.4
github.com/sirupsen/logrus v1.9.4 // indirect
github.com/spf13/afero v1.15.0 // indirect
github.com/spf13/cast v1.10.0 // indirect
@@ -84,6 +84,7 @@ require (
github.com/xdg-go/scram v1.2.0
github.com/xdg-go/stringprep v1.0.4 // indirect
github.com/youmark/pkcs8 v0.0.0-20240726163527-a2c0da244d78 // indirect
go.etcd.io/etcd/api/v3 v3.7.1
go.etcd.io/etcd/client/v3 v3.7.1
go.mongodb.org/mongo-driver v1.17.9
go.opencensus.io v0.24.0 // indirect
@@ -93,14 +94,14 @@ require (
golang.org/x/crypto v0.58.0
golang.org/x/exp v0.0.0-20260813180055-c1d0aacb2297
golang.org/x/image v0.46.0
golang.org/x/net v0.61.0
golang.org/x/net v0.62.0
golang.org/x/oauth2 v0.37.0
golang.org/x/sys v0.49.0
golang.org/x/text v0.43.0 // indirect
golang.org/x/tools v0.51.0 // indirect
golang.org/x/xerrors v0.0.0-20240903120638-7835f813f4da // indirect
google.golang.org/api v0.297.0
google.golang.org/genproto v0.0.0-20260904194346-d0f1323225a4 // indirect
google.golang.org/api v0.310.0
google.golang.org/genproto v0.0.0-20261012220534-4341edf3f941 // indirect
google.golang.org/grpc v1.86.0-dev.0.20261027044109-d619b5d02089
google.golang.org/protobuf v1.36.12
gopkg.in/inf.v0 v0.9.1 // indirect
@@ -111,7 +112,7 @@ require (
)
require (
cloud.google.com/go/kms v1.35.0
cloud.google.com/go/kms v1.36.1
github.com/Azure/azure-sdk-for-go/sdk/keyvault/azkeys v0.10.0
github.com/DATA-DOG/go-sqlmock v1.5.2
github.com/Jille/raft-grpc-transport v1.6.1
@@ -122,8 +123,7 @@ require (
github.com/apache/iceberg-go v0.7.0
github.com/apple/foundationdb/bindings/go v0.0.0-20250911184653-27f7192f47c3
github.com/arangodb/go-driver v1.6.9
github.com/armon/go-metrics v0.4.1
github.com/aws/aws-sdk-go-v2 v1.47.3
github.com/aws/aws-sdk-go-v2 v1.47.7
github.com/aws/aws-sdk-go-v2/config v1.33.6
github.com/aws/aws-sdk-go-v2/credentials v1.20.6
github.com/aws/aws-sdk-go-v2/service/s3 v1.114.1
@@ -134,6 +134,7 @@ require (
github.com/go-ldap/ldap/v3 v3.4.14
github.com/golang-jwt/jwt/v5 v5.3.1
github.com/google/flatbuffers/go v0.0.0-20230108230133-3b8644d32c50
github.com/hashicorp/go-metrics v0.7.0
github.com/hashicorp/golang-lru/v2 v2.0.7
github.com/hashicorp/raft v1.8.0
github.com/hashicorp/raft-boltdb/v2 v2.4.2
@@ -150,16 +151,16 @@ require (
github.com/redis/go-redis/v9 v9.23.0
github.com/schollz/progressbar/v3 v3.19.1
github.com/seaweedfs/go-fuse/v2 v2.9.4
github.com/shirou/gopsutil/v4 v4.26.7
github.com/shirou/gopsutil/v4 v4.26.10
github.com/tarantool/go-option v1.1.0
github.com/tarantool/go-tarantool/v3 v3.0.2
github.com/testcontainers/testcontainers-go v0.44.0
github.com/testcontainers/testcontainers-go v0.45.0
github.com/tikv/client-go/v2 v2.0.7
github.com/twmb/avro v1.9.0
github.com/xeipuuv/gojsonschema v1.2.0
github.com/ydb-platform/ydb-go-sdk-auth-environ v0.5.3
github.com/ydb-platform/ydb-go-sdk/v3 v3.155.1
go.etcd.io/etcd/client/pkg/v3 v3.7.1
go.etcd.io/etcd/client/pkg/v3 v3.7.3
go.uber.org/atomic v1.12.0
golang.org/x/sync v0.24.0
golang.org/x/tools/godoc v0.1.0-deprecated
@@ -169,12 +170,12 @@ require (
require github.com/k0kubun/colorstring v0.0.0-20150214042306-9440f1994b88 // indirect
require (
cloud.google.com/go/longrunning v1.2.0 // indirect
cloud.google.com/go/longrunning v1.3.0 // indirect
cloud.google.com/go/pubsub/v2 v2.7.0 // indirect
dario.cat/mergo v1.0.2 // indirect
github.com/Azure/azure-sdk-for-go/sdk/keyvault/internal v0.7.1 // indirect
github.com/Azure/azure-sdk-for-go/sdk/storage/internal v0.1.0 // indirect
github.com/Azure/go-ansiterm v0.0.0-20250102033503-faa5f7b0171c // indirect
github.com/Azure/go-ansiterm v0.0.0-20261006220739-8c912ac31dd4 // indirect
github.com/FilenCloudDienste/filen-sdk-go v0.0.39 // indirect
github.com/ProtonMail/gopenpgp/v3 v3.4.1 // indirect
github.com/RoaringBitmap/roaring/v2 v2.26.0 // indirect
@@ -206,7 +207,7 @@ require (
github.com/dave/dst v0.27.2 // indirect
github.com/diskfs/go-diskfs v1.9.4 // indirect
github.com/distribution/reference v0.6.0 // indirect
github.com/docker/go-connections v0.7.0 // indirect
github.com/docker/go-connections v0.8.2 // indirect
github.com/docker/go-units v0.5.0 // indirect
github.com/dromara/dongle v1.0.1 // indirect
github.com/geoarrow/geoarrow-go v0.0.0-20260403143023-f54751c3e3a1 // indirect
@@ -235,11 +236,11 @@ require (
github.com/lib/pq v1.12.3 // indirect
github.com/lithammer/shortuuid/v3 v3.0.7 // indirect
github.com/lpar/calendar v0.2.0 // indirect
github.com/magiconair/properties v1.8.10 // indirect
github.com/magiconair/properties v1.18.12 // indirect
github.com/moby/docker-image-spec v1.3.1 // indirect
github.com/moby/go-archive v0.3.0 // indirect
github.com/moby/moby/api v1.55.0 // indirect
github.com/moby/moby/client v0.5.0 // indirect
github.com/moby/go-archive v0.3.3 // indirect
github.com/moby/moby/api v1.56.1 // indirect
github.com/moby/moby/client v0.6.2 // indirect
github.com/moby/patternmatcher v0.6.1 // indirect
github.com/moby/sys/sequential v0.7.0 // indirect
github.com/moby/sys/user v0.4.1 // indirect
@@ -278,23 +279,24 @@ require (
go.opentelemetry.io/otel/exporters/otlp/otlptrace v1.46.0 // indirect
go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc v1.46.0 // indirect
go.opentelemetry.io/otel/exporters/zipkin v1.45.0 // indirect
go.opentelemetry.io/otel/log v1.47.0 // indirect
go.opentelemetry.io/proto/otlp v1.11.0 // indirect
go.uber.org/mock v0.5.2 // indirect
go.yaml.in/yaml/v2 v2.4.4 // indirect
go.yaml.in/yaml/v3 v3.0.5 // indirect
golang.org/x/mod v0.41.0 // indirect
go.yaml.in/yaml/v3 v3.0.6 // indirect
golang.org/x/mod v0.42.0 // indirect
gonum.org/v1/gonum v0.17.0 // indirect
)
require (
cel.dev/expr v0.25.3 // indirect
cloud.google.com/go/auth v0.23.2 // indirect
cloud.google.com/go/auth/oauth2adapt v0.2.8 // indirect
cloud.google.com/go/auth v0.24.1-0.20261001053825-dbc26066f70a // indirect
cloud.google.com/go/auth/oauth2adapt v0.3.0 // indirect
cloud.google.com/go/compute/metadata v0.10.0 // indirect
cloud.google.com/go/iam v1.13.0 // indirect
cloud.google.com/go/iam v1.14.0 // indirect
cloud.google.com/go/monitoring v1.30.0 // indirect
filippo.io/edwards25519 v1.2.0 // indirect
github.com/Azure/azure-sdk-for-go/sdk/azcore v1.23.3
github.com/Azure/azure-sdk-for-go/sdk/azcore v1.24.0
github.com/Azure/azure-sdk-for-go/sdk/azidentity v1.14.1
github.com/Azure/azure-sdk-for-go/sdk/internal v1.13.0 // indirect
github.com/Azure/azure-sdk-for-go/sdk/storage/azblob v1.8.2
@@ -321,19 +323,19 @@ require (
github.com/aws/aws-sdk-go-v2/aws/protocol/eventstream v1.7.20 // indirect
github.com/aws/aws-sdk-go-v2/feature/ec2/imds v1.20.1 // indirect
github.com/aws/aws-sdk-go-v2/feature/s3/manager v1.22.34 // indirect
github.com/aws/aws-sdk-go-v2/internal/configsources v1.5.4 // indirect
github.com/aws/aws-sdk-go-v2/internal/endpoints/v2 v2.8.4 // indirect
github.com/aws/aws-sdk-go-v2/internal/v4a v1.5.4 // indirect
github.com/aws/aws-sdk-go-v2/service/internal/accept-encoding v1.13.19 // indirect
github.com/aws/aws-sdk-go-v2/internal/configsources v1.5.8 // indirect
github.com/aws/aws-sdk-go-v2/internal/endpoints/v2 v2.8.8 // indirect
github.com/aws/aws-sdk-go-v2/internal/v4a v1.5.8 // indirect
github.com/aws/aws-sdk-go-v2/service/internal/accept-encoding v1.13.23 // indirect
github.com/aws/aws-sdk-go-v2/service/internal/checksum v1.11.5 // indirect
github.com/aws/aws-sdk-go-v2/service/internal/presigned-url v1.14.4 // indirect
github.com/aws/aws-sdk-go-v2/service/internal/presigned-url v1.14.8 // indirect
github.com/aws/aws-sdk-go-v2/service/internal/s3shared v1.20.4 // indirect
github.com/aws/aws-sdk-go-v2/service/sns v1.46.0 // indirect
github.com/aws/aws-sdk-go-v2/service/sqs v1.51.0 // indirect
github.com/aws/aws-sdk-go-v2/service/sso v1.38.1 // indirect
github.com/aws/aws-sdk-go-v2/service/ssooidc v1.43.1 // indirect
github.com/aws/aws-sdk-go-v2/service/sts v1.51.1
github.com/aws/smithy-go v1.28.5
github.com/aws/aws-sdk-go-v2/service/sts v1.51.6
github.com/aws/smithy-go v1.28.8
github.com/boltdb/bolt v1.3.1 // indirect
github.com/bradenaw/juniper v0.15.3 // indirect
github.com/buengese/sgzip v0.1.1 // indirect
@@ -351,7 +353,7 @@ require (
github.com/d4l3k/messagediff v1.2.1 // indirect
github.com/dgryski/go-farm v0.0.0-20200201041132-a6ae2369ad13 // indirect
github.com/dropbox/dropbox-sdk-go-unofficial/v6 v6.4.0 // indirect
github.com/ebitengine/purego v0.10.2 // indirect
github.com/ebitengine/purego v0.11.2 // indirect
github.com/elastic/gosigar v0.14.3 // indirect
github.com/emersion/go-message v0.18.2 // indirect
github.com/emersion/go-vcard v0.0.0-20260618161152-d854b7e0e2d3 // indirect
@@ -379,8 +381,8 @@ require (
github.com/gofrs/flock v0.13.0 // indirect
github.com/gogo/protobuf v1.3.2 // indirect
github.com/golang-jwt/jwt/v4 v4.5.2 // indirect
github.com/google/s2a-go v0.1.9 // indirect
github.com/googleapis/enterprise-certificate-proxy v0.3.21 // indirect
github.com/google/s2a-go v0.1.11 // indirect
github.com/googleapis/enterprise-certificate-proxy v0.3.24 // indirect
github.com/gorilla/schema v1.4.1 // indirect
github.com/gorilla/securecookie v1.1.2 // indirect
github.com/gorilla/sessions v1.4.0
@@ -389,7 +391,6 @@ require (
github.com/hashicorp/go-cleanhttp v0.5.2 // indirect
github.com/hashicorp/go-hclog v1.6.3 // indirect
github.com/hashicorp/go-immutable-radix v1.3.1 // indirect
github.com/hashicorp/go-metrics v0.7.0 // indirect
github.com/hashicorp/go-msgpack/v2 v2.1.5 // indirect
github.com/hashicorp/go-retryablehttp v0.7.8 // indirect
github.com/hashicorp/golang-lru v1.0.2 // indirect
@@ -412,7 +413,7 @@ require (
github.com/leodido/go-urn v1.4.0 // indirect
github.com/lufia/plan9stats v0.0.0-20260627054121-477a66015f15 // indirect
github.com/mattn/go-colorable v0.1.15 // indirect
github.com/mattn/go-runewidth v0.0.24 // indirect
github.com/mattn/go-runewidth v0.0.31 // indirect
github.com/mitchellh/colorstring v0.0.0-20190213212951-d06e56a500db // indirect
github.com/mitchellh/go-homedir v1.1.0 // indirect
github.com/mitchellh/mapstructure v1.5.1-0.20220423185008-bf980b35cac4
@@ -441,7 +442,7 @@ require (
github.com/pkg/browser v0.0.0-20240102092130-5ac0b6a4141c // indirect
github.com/pkg/xattr v0.4.12 // indirect
github.com/planetscale/vtprotobuf v0.6.1-0.20240319094008-0393e58bdf10
github.com/power-devops/perfstat v0.0.0-20240221224432-82ca36839d55 // indirect
github.com/power-devops/perfstat v0.0.0-20260805114148-88456608a4f6 // indirect
github.com/putdotio/go-putio/putio v0.0.0-20200123120452-16d982cac2b8 // indirect
github.com/relvacode/iso8601 v1.7.0 // indirect
github.com/rfjakob/eme v1.2.0 // indirect
@@ -479,22 +480,21 @@ require (
github.com/zeebo/blake3 v0.2.4 // indirect
github.com/zeebo/errs v1.4.0 // indirect
go.etcd.io/bbolt v1.5.0 // indirect
go.etcd.io/etcd/api/v3 v3.7.1 // indirect
go.opentelemetry.io/auto/sdk v1.2.1 // indirect
go.opentelemetry.io/contrib/detectors/gcp v1.46.0 // indirect
go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc v0.71.0 // indirect
go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp v0.71.0 // indirect
go.opentelemetry.io/otel v1.46.0 // indirect
go.opentelemetry.io/otel/metric v1.46.0 // indirect
go.opentelemetry.io/otel/sdk v1.46.0 // indirect
go.opentelemetry.io/otel/sdk/metric v1.46.0 // indirect
go.opentelemetry.io/otel/trace v1.46.0 // indirect
go.opentelemetry.io/otel v1.47.0 // indirect
go.opentelemetry.io/otel/metric v1.47.0 // indirect
go.opentelemetry.io/otel/sdk v1.47.0 // indirect
go.opentelemetry.io/otel/sdk/metric v1.47.0 // indirect
go.opentelemetry.io/otel/trace v1.47.0 // indirect
go.uber.org/multierr v1.11.0 // indirect
go.uber.org/zap v1.28.0 // indirect
golang.org/x/term v0.47.0
golang.org/x/time v0.16.0
google.golang.org/genproto/googleapis/api v0.0.0-20260921155816-b14227669459 // indirect
google.golang.org/genproto/googleapis/rpc v0.0.0-20260921155816-b14227669459 // indirect
google.golang.org/genproto/googleapis/api v0.0.0-20261012220534-4341edf3f941 // indirect
google.golang.org/genproto/googleapis/rpc v0.0.0-20261019180522-ce908d29299b
gopkg.in/natefinch/lumberjack.v2 v2.2.1 // indirect
gopkg.in/validator.v2 v2.0.1 // indirect
gopkg.in/yaml.v2 v2.4.0 // indirect
+96 -96
View File
@@ -4,20 +4,20 @@ cloud.google.com/go v0.26.0/go.mod h1:aQUYkXzVsufM+DwF1aE+0xfcU+56JwCaLick0ClmMT
cloud.google.com/go v0.34.0/go.mod h1:aQUYkXzVsufM+DwF1aE+0xfcU+56JwCaLick0ClmMTw=
cloud.google.com/go v0.123.0 h1:2NAUJwPR47q+E35uaJeYoNhuNEM9kM8SjgRgdeOJUSE=
cloud.google.com/go v0.123.0/go.mod h1:xBoMV08QcqUGuPW65Qfm1o9Y4zKZBpGS+7bImXLTAZU=
cloud.google.com/go/auth v0.23.2 h1:pxSCpfiji41hpzpPdMCftEUCezpgpqmmDdYiAjCKXxo=
cloud.google.com/go/auth v0.23.2/go.mod h1:4DhBRcqvtljQN3dJ57qtqbib5ZGCYE5f2crfiiC2EM0=
cloud.google.com/go/auth/oauth2adapt v0.2.8 h1:keo8NaayQZ6wimpNSmW5OPc283g65QNIiLpZnkHRbnc=
cloud.google.com/go/auth/oauth2adapt v0.2.8/go.mod h1:XQ9y31RkqZCcwJWNSx2Xvric3RrU88hAYYbjDWYDL+c=
cloud.google.com/go/auth v0.24.1-0.20261001053825-dbc26066f70a h1:jW1ZLFy9oa5ai8LOI40z6s+DL1cO/L1KSm9hVGnibn8=
cloud.google.com/go/auth v0.24.1-0.20261001053825-dbc26066f70a/go.mod h1:1dmBXa9xfBvBr39ZnDUDkaaA91ccq9CcqENDgJA4d7c=
cloud.google.com/go/auth/oauth2adapt v0.3.0 h1:FY8oSZpCYoUNv6QxVODuMjQz4IlSOVeiQtZ08vLPz88=
cloud.google.com/go/auth/oauth2adapt v0.3.0/go.mod h1:7+2uCm7++XFO+/lN06c2HXpDXb/NMNn2/UwyBPbTnkk=
cloud.google.com/go/compute/metadata v0.10.0 h1:pyKMUQSwchgkIBBJGdILqQbs/BNJXqwSA7Ej6LAvvtY=
cloud.google.com/go/compute/metadata v0.10.0/go.mod h1:rGFHRrIif570kSibjFTMbt6/4/tzgJWFGI/HVol4GIk=
cloud.google.com/go/iam v1.13.0 h1:ufT3FPT5rFFXu6UtLkNoxaOaV5EuA1dsSkmemCSTo6U=
cloud.google.com/go/iam v1.13.0/go.mod h1:gHXdDEiPDvqd1q1KwBDGQlgZY/BwY760zU2LhOZS5w0=
cloud.google.com/go/kms v1.35.0 h1:nJ/ktaqspx1nPM9vIcO0SHbhqCAm8nvAxL1siuVgKm0=
cloud.google.com/go/kms v1.35.0/go.mod h1:0++71pIHvJL+GmMa8K4jOWFq7gNOX3jm2PRMSJwTKJw=
cloud.google.com/go/logging v1.19.1 h1:7SsLhyTDBDrJw+Ll6Ns3I2mByqHXvJUc3rGjSlwiWgU=
cloud.google.com/go/logging v1.19.1/go.mod h1:2IkQ/d8jVJqV2qW8ZUGUiMjdZG1gkLD2JReGbZ8isqg=
cloud.google.com/go/longrunning v1.2.0 h1:WjYH3YHBGCxGJP9M4dWGHBfXr/cFIjMkNgWcJj7/iMM=
cloud.google.com/go/longrunning v1.2.0/go.mod h1:5KMQALFGOCtFoi2xSOA1u3H7WKlhmckgiyFw7+LGQp0=
cloud.google.com/go/iam v1.14.0 h1:YRIB1/4xxwJ+NfI6RQr1WaOoHHeV0MizJEj940nBXJA=
cloud.google.com/go/iam v1.14.0/go.mod h1:JTPZfbHdGXJrOYREtkRoFzCDGaBdbA7ILLKO/DDuav0=
cloud.google.com/go/kms v1.36.1 h1:jIXDoN/lxjVt/zc219tIwqAWPVWVzcLJ7s6Oq3n0Y5U=
cloud.google.com/go/kms v1.36.1/go.mod h1:1oVAwTvmgswE8PiQeyffdNxpRLw2F0UfQlLwTrYlRNg=
cloud.google.com/go/logging v1.20.0 h1:l7yzCB8/JSY5PHT9RsaqdT1jaBGbM6KlSJDX210HlXM=
cloud.google.com/go/logging v1.20.0/go.mod h1:ALUaJQZM691Eeujy9kFx0xREq3AMPuGd+pTRZuTYv+w=
cloud.google.com/go/longrunning v1.3.0 h1:lZ6c8SJJBmILp/Wnr3vGnBLa0q+cYfSHRCcUfVI2hpI=
cloud.google.com/go/longrunning v1.3.0/go.mod h1:wpgLxCn+UA3/TZ1bj/fW39mfDpwVlMPu1xKJyoMt7V0=
cloud.google.com/go/monitoring v1.30.0 h1:r/d+JUbyKmJ8b07iznuKfzVzrIXTWxHQ3lBRm3x2LlY=
cloud.google.com/go/monitoring v1.30.0/go.mod h1:htlUR0QWVMrjFzZmN4LGnMAve9xB/eduwjmINxVZ8RM=
cloud.google.com/go/pubsub v1.51.1 h1:R3G1wCOxBO7jRpL8x2pdZMv1GAJDF6ax/m2zPOtvTNE=
@@ -34,8 +34,8 @@ filippo.io/edwards25519 v1.2.0 h1:crnVqOiS4jqYleHd9vaKZ+HKtHfllngJIiOpNpoJsjo=
filippo.io/edwards25519 v1.2.0/go.mod h1:xzAOLCNug/yB62zG1bQ8uziwrIqIuxhctzJT18Q77mc=
github.com/AdaLogics/go-fuzz-headers v0.0.0-20240806141605-e8a1dd7889d6 h1:He8afgbRMd7mFxO99hRNu+6tazq8nFF9lIwo9JFroBk=
github.com/AdaLogics/go-fuzz-headers v0.0.0-20240806141605-e8a1dd7889d6/go.mod h1:8o94RPi1/7XTJvwPpRSzSUedZrtlirdB3r9Z20bi2f8=
github.com/Azure/azure-sdk-for-go/sdk/azcore v1.23.3 h1:qgnWTiWs1G7AqcKZegFvkJD1bmXJcaDU3uC0ELSgZ5U=
github.com/Azure/azure-sdk-for-go/sdk/azcore v1.23.3/go.mod h1:XtxD48QPInqoj+GMlcRlXHEYsoOxFnP6w2JgIZ9rDS4=
github.com/Azure/azure-sdk-for-go/sdk/azcore v1.24.0 h1:5H6YRLGMoC8z5aXpxqcD6FfswI8mwoVNHSecvIoazt4=
github.com/Azure/azure-sdk-for-go/sdk/azcore v1.24.0/go.mod h1:Sk63W6X/TElfcE2BknMYg0Q9JuCpebgjnHcbs5md4NM=
github.com/Azure/azure-sdk-for-go/sdk/azidentity v1.14.1 h1:u93s+zU2JD62im61Bm5CZIc1ZrOJaIAWEg0WOrMVkEo=
github.com/Azure/azure-sdk-for-go/sdk/azidentity v1.14.1/go.mod h1:oXtinPO4OLj9d1DOTrqrL1oRwGhcqadvAmrl6wTeGlk=
github.com/Azure/azure-sdk-for-go/sdk/azidentity/cache v0.4.0 h1:xFaZZ+IubdftrDHnGGwZ6QvQ3KHTtWl2MCK+GMt2vxs=
@@ -54,8 +54,8 @@ github.com/Azure/azure-sdk-for-go/sdk/storage/azfile v1.7.0 h1:cuiKf1UVyWHu+XSQg
github.com/Azure/azure-sdk-for-go/sdk/storage/azfile v1.7.0/go.mod h1:9JSyvgXLPAOC4jfhgZg58XeU2FJHsGmxuYSclCFQ4ZY=
github.com/Azure/azure-sdk-for-go/sdk/storage/internal v0.1.0 h1:X/bTwe0qpvmvPPutc+LWo7T4cKMqDP9JHP4k+Co9ZB4=
github.com/Azure/azure-sdk-for-go/sdk/storage/internal v0.1.0/go.mod h1:+kX7ahyFelQ42F3RScM2CFgHJljp0phc/FieKLYVpFI=
github.com/Azure/go-ansiterm v0.0.0-20250102033503-faa5f7b0171c h1:udKWzYgxTojEKWjV8V+WSxDXJ4NFATAsZjh8iIbsQIg=
github.com/Azure/go-ansiterm v0.0.0-20250102033503-faa5f7b0171c/go.mod h1:xomTg63KZ2rFqZQzSB4Vz2SUXa1BpHTVz9L5PTmPC4E=
github.com/Azure/go-ansiterm v0.0.0-20261006220739-8c912ac31dd4 h1:m2xcjRABtG288vnlV3U/PKh/HnPURxZp50RKNUwBxM8=
github.com/Azure/go-ansiterm v0.0.0-20261006220739-8c912ac31dd4/go.mod h1:MTUE7ZuZO89FAtAwozGg6AWD83pcdefnEq5gF82MB8s=
github.com/Azure/go-ntlmssp v0.1.1 h1:l+FM/EEMb0U9QZE7mKNEDw5Mu3mFiaa2GKOoTSsNDPw=
github.com/Azure/go-ntlmssp v0.1.1/go.mod h1:NYqdhxd/8aAct/s4qSYZEerdPuH1liG2/X9DiVTbhpk=
github.com/AzureAD/microsoft-authentication-extensions-for-go/cache v0.1.1 h1:WJTmL004Abzc5wDB5VtZG2PJk5ndYDgVacGqfirKxjM=
@@ -173,8 +173,8 @@ github.com/armon/go-metrics v0.4.1 h1:hR91U9KYmb6bLBYLQjyM+3j+rcd/UhE+G78SFnF8gJ
github.com/armon/go-metrics v0.4.1/go.mod h1:E6amYzXo6aW1tqzoZGT755KkbgrJsSdpwZ+3JqfkOG4=
github.com/aws/aws-sdk-go v1.55.8 h1:JRmEUbU52aJQZ2AjX4q4Wu7t4uZjOu71uyNmaWlUkJQ=
github.com/aws/aws-sdk-go v1.55.8/go.mod h1:ZkViS9AqA6otK+JBBNH2++sx1sgxrPKcSzPPvQkUtXk=
github.com/aws/aws-sdk-go-v2 v1.47.3 h1:HRoT/MkNi3bjO5xF4dCzLzvVmkvRGHj3/cLINDnJejk=
github.com/aws/aws-sdk-go-v2 v1.47.3/go.mod h1:xSl+OSI5ZUw6bSMupWc218zE5Pl+65MCCu3ySWFY99A=
github.com/aws/aws-sdk-go-v2 v1.47.7 h1:jBmtzRtGWVAzbfc4RBWo1dTte0ZZH8p9U42YJIw2+Bw=
github.com/aws/aws-sdk-go-v2 v1.47.7/go.mod h1:I8lFSH4MGrZiw2m7miCrRYHsgpC1+HW8uiVGOiPYPcw=
github.com/aws/aws-sdk-go-v2/aws/protocol/eventstream v1.7.20 h1:GPRlPwz40I2B2VrBEASOA3Bi77NyeqejNLkifosX0rs=
github.com/aws/aws-sdk-go-v2/aws/protocol/eventstream v1.7.20/go.mod h1:g7PNzKcsOKWb4fkSRBA7BZVAS6Y8IcxzN+nRohhQ1Q8=
github.com/aws/aws-sdk-go-v2/config v1.33.6 h1:MBjkSTLczek/UgiK+EYPIoRTqE7gP8vtW3OFbFo7Nug=
@@ -187,18 +187,18 @@ github.com/aws/aws-sdk-go-v2/feature/s3/manager v1.22.34 h1:Pn7OsMwBLbkZ6OnCxWHA
github.com/aws/aws-sdk-go-v2/feature/s3/manager v1.22.34/go.mod h1:eToXR/Gk1uqpn04eSmdgVXwfS0WvH8aG4eBFr8ygbpU=
github.com/aws/aws-sdk-go-v2/feature/s3/transfermanager v0.4.3 h1:vsFkmFrfzODkxVbyuPLscv5j5lC8KbG48LZwV+p56Y0=
github.com/aws/aws-sdk-go-v2/feature/s3/transfermanager v0.4.3/go.mod h1:pjQYVtvVBEtlsf2p6nXMsjPuDaWs9GM44FMdHbmnmYU=
github.com/aws/aws-sdk-go-v2/internal/configsources v1.5.4 h1:CLq4+8UHCI+ZZYl/EuJxXovaIVN2xeeT8JV+dsApQ5E=
github.com/aws/aws-sdk-go-v2/internal/configsources v1.5.4/go.mod h1:Wv4q5sAM04xAMkoOedxLx2inVf6K5FdxYp+A61L+q/0=
github.com/aws/aws-sdk-go-v2/internal/endpoints/v2 v2.8.4 h1:dD4MR81I7YkpEBRk6UP9rocC2QnT3qVuXwzlYTtfGEs=
github.com/aws/aws-sdk-go-v2/internal/endpoints/v2 v2.8.4/go.mod h1:EcXV1kAFd5XwSkDHlj94gnF3q5CkJyYiIJfH8N0VmrE=
github.com/aws/aws-sdk-go-v2/internal/v4a v1.5.4 h1:7Wo47d/xn/7KttCSBd8EGYeZ7ULRFRkUHr6vkZPBzVQ=
github.com/aws/aws-sdk-go-v2/internal/v4a v1.5.4/go.mod h1:tDB2IVC1xC3vX8o+6uRlzhTxP3g1b77CZXFX/oD2FnQ=
github.com/aws/aws-sdk-go-v2/service/internal/accept-encoding v1.13.19 h1:bAdDl/HkGCcGPoe25ToSHEw23VIxt6CT5fLcg111BKg=
github.com/aws/aws-sdk-go-v2/service/internal/accept-encoding v1.13.19/go.mod h1:KaUzbLxv4CeSxh6ZCl9B4m7CuFenS8kUEaDs+f/DQr4=
github.com/aws/aws-sdk-go-v2/internal/configsources v1.5.8 h1:gzIWc3DTPfPbWKrQnmL6BV6WD7NPD1htxzgdk/bQ/Z4=
github.com/aws/aws-sdk-go-v2/internal/configsources v1.5.8/go.mod h1:vvUA5cDAZWGo4R14FZPZfXfJCmgsafOWEq78Kg0D+ug=
github.com/aws/aws-sdk-go-v2/internal/endpoints/v2 v2.8.8 h1:PWv+vG8aCLS8qd+HPdYhK7Vees90aKHWFFQ1AGYD/8E=
github.com/aws/aws-sdk-go-v2/internal/endpoints/v2 v2.8.8/go.mod h1:xxPZEpflYHJA12366ZGyq66z9dHPCoeCIu99Jck+MPw=
github.com/aws/aws-sdk-go-v2/internal/v4a v1.5.8 h1:WOp5De9iWdEHDPh2Kb0tBP3vGBm/dVAuRGh4a4tqduo=
github.com/aws/aws-sdk-go-v2/internal/v4a v1.5.8/go.mod h1:z8W8YgfhbWl62XqcTtK+CxPrE+iHGRVUOtrrbIOp7ZY=
github.com/aws/aws-sdk-go-v2/service/internal/accept-encoding v1.13.23 h1:MCIR6iZaCKO4UE0hUvoWWLI6JVjppopV64JfAJaMjng=
github.com/aws/aws-sdk-go-v2/service/internal/accept-encoding v1.13.23/go.mod h1:pphWFS+z0MmIPivj8c/6bATd1Z+Av7Ust75DPdEFsPA=
github.com/aws/aws-sdk-go-v2/service/internal/checksum v1.11.5 h1:/TYsZXdA8UTa+WCtCYSAJIr1vwl0+eho6TUgJGwFFO8=
github.com/aws/aws-sdk-go-v2/service/internal/checksum v1.11.5/go.mod h1:qPqp1Uwd/BqdhPufv6oem9j5J7HNsgc2V22dUiDPn+s=
github.com/aws/aws-sdk-go-v2/service/internal/presigned-url v1.14.4 h1:29SvnfGhXjTl8ONxFwbj2rs6lbhiFXD2CgFQmbT/bXY=
github.com/aws/aws-sdk-go-v2/service/internal/presigned-url v1.14.4/go.mod h1:wm04I5DMuNVvZHFe/dHnUxincvNbbK7AiNBbYsQivek=
github.com/aws/aws-sdk-go-v2/service/internal/presigned-url v1.14.8 h1:+PMpWlJYOn2tm7sm6OZ/ZLd4jStl17T7wjNapd1Rczs=
github.com/aws/aws-sdk-go-v2/service/internal/presigned-url v1.14.8/go.mod h1:cnG3xZLGQL4c/bt8pjdzlFtZsW7epBBuNS04a5xMjdo=
github.com/aws/aws-sdk-go-v2/service/internal/s3shared v1.20.4 h1:pPiWfgeNxqluKEph7hvU88kuGKBPOWzO+Dk9t2zqqNs=
github.com/aws/aws-sdk-go-v2/service/internal/s3shared v1.20.4/go.mod h1:YlwGoIUDG/3kBQbdNOVs/xKZ9J01G8e/6D1mRBj9uTk=
github.com/aws/aws-sdk-go-v2/service/s3 v1.114.1 h1:9T6eO8/WXhij4PQaa2WaYsdekSCmT9Hw66Pw0S8do0Q=
@@ -213,10 +213,10 @@ github.com/aws/aws-sdk-go-v2/service/sso v1.38.1 h1:Umtl/0YZhng4xndfW3lKJrYYP7NL
github.com/aws/aws-sdk-go-v2/service/sso v1.38.1/go.mod h1:rRD/dnm7q0HYE/I5TMaPgkWyyUGLcwuxHLABsLnQ3e0=
github.com/aws/aws-sdk-go-v2/service/ssooidc v1.43.1 h1:orIWdNiLgzrhu/11RcPPKO/SBzUUymbUQuZbSPImghg=
github.com/aws/aws-sdk-go-v2/service/ssooidc v1.43.1/go.mod h1:skwM/xsbR/1ReUTesv9BhpJp1VjajR7DWQnuVLwiXsQ=
github.com/aws/aws-sdk-go-v2/service/sts v1.51.1 h1:0HOqZXRvMytH6bFHVIc0oJX07sZjfhz0zXtjs6gdE8s=
github.com/aws/aws-sdk-go-v2/service/sts v1.51.1/go.mod h1:26zA0GhDrLo+yiLI2yXWxqB1PdsShfLikoI7GOEgugM=
github.com/aws/smithy-go v1.28.5 h1:OqbqKtzY9NUnr2yAe9+hiTdFb20tNaK/lI0PMOGXX9k=
github.com/aws/smithy-go v1.28.5/go.mod h1:YE2RhdIuDbA5E5bTdciG9KrW3+TiEONeUWCqxX9i1Fc=
github.com/aws/aws-sdk-go-v2/service/sts v1.51.6 h1:BVjKrtRUtSYKztBCCfE1oAB+d3ViKcW2cKZfserOqMQ=
github.com/aws/aws-sdk-go-v2/service/sts v1.51.6/go.mod h1:eSukFLeLkPMm+1OxDR+F0joauZdyBkF47Ow7MqjZiJw=
github.com/aws/smithy-go v1.28.8 h1:aEmdAq7gVCoGCnvf1NnELlu3I6D/nhqrC321FyFiQkQ=
github.com/aws/smithy-go v1.28.8/go.mod h1:YE2RhdIuDbA5E5bTdciG9KrW3+TiEONeUWCqxX9i1Fc=
github.com/bahlo/generic-list-go v0.2.0 h1:5sz/EEAK+ls5wF+NeqDpk5+iNdMDXrh3z3nPnH1Wvgk=
github.com/bahlo/generic-list-go v0.2.0/go.mod h1:2KvAjgMlE5NNynlg/5iLrrCCZ2+5xWbdbCW3pNTGyYg=
github.com/bazelbuild/rules_go v0.46.0 h1:CTefzjN/D3Cdn3rkrM6qMWuQj59OBcuOjyIp3m4hZ7s=
@@ -389,8 +389,8 @@ github.com/docker/docker v28.5.2+incompatible h1:DBX0Y0zAjZbSrm1uzOkdr1onVghKaft
github.com/docker/docker v28.5.2+incompatible/go.mod h1:eEKB0N0r5NX/I1kEveEz05bcu8tLC/8azJZsviup8Sk=
github.com/docker/docker-credential-helpers v0.9.8 h1:bIREROb7So6PRlq6KTtdS9MPEjC29OQRkFNlvK2OX8Q=
github.com/docker/docker-credential-helpers v0.9.8/go.mod h1:v1S+hepowrQXITkEfw6o4+BMbGot02wiKpzWhGUZK6c=
github.com/docker/go-connections v0.7.0 h1:6SsRfJddP22WMrCkj19x9WKjEDTB+ahsdiGYf0mN39c=
github.com/docker/go-connections v0.7.0/go.mod h1:no1qkHdjq7kLMGUXYAduOhYPSJxxvgWBh7ogVvptn3Q=
github.com/docker/go-connections v0.8.2 h1:9d9DEokx5W7qlr0shkFXIxEA7+TTD+llwzcykSlxMNc=
github.com/docker/go-connections v0.8.2/go.mod h1:no1qkHdjq7kLMGUXYAduOhYPSJxxvgWBh7ogVvptn3Q=
github.com/docker/go-units v0.5.0 h1:69rxXcBk27SvSaaxTtLh/8llcHD8vYHT7WSdRZ/jvr4=
github.com/docker/go-units v0.5.0/go.mod h1:fgPhTUdO+D/Jk86RDLlptpiXQzgHJF7gydDDbaIK4Dk=
github.com/dromara/dongle v1.0.1 h1:si/7UP/EXxnFVZok1cNos70GiMGxInAYMilHQFP5dJs=
@@ -407,8 +407,8 @@ github.com/eapache/go-xerial-snappy v0.0.0-20230731223053-c322873962e3 h1:Oy0F4A
github.com/eapache/go-xerial-snappy v0.0.0-20230731223053-c322873962e3/go.mod h1:YvSRo5mw33fLEx1+DlK6L2VV43tJt5Eyel9n9XBcR+0=
github.com/eapache/queue v1.1.0 h1:YOEu7KNc61ntiQlcEeUIoDTJ2o8mQznoNvUhiigpIqc=
github.com/eapache/queue v1.1.0/go.mod h1:6eCeP0CKFpHLu8blIFXhExK/dRa7WDZfr6jVFPTqq+I=
github.com/ebitengine/purego v0.10.2 h1:W809HbnvzAxgdm+aOvlSekrM16wGCdT/e76+9tS7gzE=
github.com/ebitengine/purego v0.10.2/go.mod h1:iIjxzd6CiRiOG0UyXP+V1+jWqUXVjPKLAI0mRfJZTmQ=
github.com/ebitengine/purego v0.11.2 h1:WWiTx8rD8zYdOwSj8O/NiNbsOvSAgJMTlyV1XlbUSeU=
github.com/ebitengine/purego v0.11.2/go.mod h1:DCHPP08djqhNSoTfImcnHYQRZmd0qhakvrozqaEYhGQ=
github.com/eiannone/keyboard v0.0.0-20220611211555-0d226195f203 h1:XBBHcIb256gUJtLmY22n99HaZTz+r2Z51xUPi01m3wg=
github.com/eiannone/keyboard v0.0.0-20220611211555-0d226195f203/go.mod h1:E1jcSv8FaEny+OP/5k9UxZVw9YFWGj7eI4KR/iOBqCg=
github.com/elastic/gosigar v0.14.3 h1:xwkKwPia+hSfg9GqrCUKYdId102m9qTJIIr7egmK/uo=
@@ -619,8 +619,8 @@ github.com/google/martian/v3 v3.3.3 h1:DIhPTQrbPkgs2yJYdXU/eNACCG5DVQjySNRNlflZ9
github.com/google/martian/v3 v3.3.3/go.mod h1:iEPrYcgCF7jA9OtScMFQyAlZZ4YXTKEtJ1E6RWzmBA0=
github.com/google/pprof v0.0.0-20260802141513-ef3492d7dac3 h1:LMLX+LgTNWpfvCBdFebv6EsYotImrt/Ppc5cXIriCSo=
github.com/google/pprof v0.0.0-20260802141513-ef3492d7dac3/go.mod h1:jl5iWTm0/hd5PjEYEOuwAJ57L/CibdZfrqZ5XA5GrCk=
github.com/google/s2a-go v0.1.9 h1:LGD7gtMgezd8a/Xak7mEWL0PjoTQFvpRudN895yqKW0=
github.com/google/s2a-go v0.1.9/go.mod h1:YA0Ei2ZQL3acow2O62kdp9UlnvMmU7kA6Eutn0dXayM=
github.com/google/s2a-go v0.1.11 h1:7v1V1TraNGP63WptsAkNqr2w8jgm4vM6A6qx6Lsw9iM=
github.com/google/s2a-go v0.1.11/go.mod h1:pz4tyvwXvJLLbyrkh6FW1eS2zPUXMaTmyNhYtyP2tNw=
github.com/google/shlex v0.0.0-20191202100458-e7afc7fbc510 h1:El6M4kTTCOh6aBiKaUGG7oYTSPP8MxqL4YI3kZKwcP4=
github.com/google/shlex v0.0.0-20191202100458-e7afc7fbc510/go.mod h1:pupxD2MaaD3pAXIBCelhxNneeOaAeabZDe5s4K6zSpQ=
github.com/google/uuid v1.1.2/go.mod h1:TIyPZe4MgqvfeYDBFedMoGGpEw/LqOeaOT+nhxU+yHo=
@@ -629,10 +629,10 @@ github.com/google/uuid v1.6.0 h1:NIvaJDMOsjHA8n1jAhLSgzrAzy1Hgr+hNrb57e+94F0=
github.com/google/uuid v1.6.0/go.mod h1:TIyPZe4MgqvfeYDBFedMoGGpEw/LqOeaOT+nhxU+yHo=
github.com/google/wire v0.7.0 h1:JxUKI6+CVBgCO2WToKy/nQk0sS+amI9z9EjVmdaocj4=
github.com/google/wire v0.7.0/go.mod h1:n6YbUQD9cPKTnHXEBN2DXlOp/mVADhVErcMFb0v3J18=
github.com/googleapis/enterprise-certificate-proxy v0.3.21 h1:OFdQ3tnCX/zaQ0Cedur3D3z7kI6HiLX9g3TiAN4/DFU=
github.com/googleapis/enterprise-certificate-proxy v0.3.21/go.mod h1:L3D/IQExI6LqEjBdXcZQ1WluSgigQmSwBboFstVPM4w=
github.com/googleapis/gax-go/v2 v2.24.1 h1:AtqTN21IXMMWo99LiEVAiBfNNQmO40d8xUfZI640mc0=
github.com/googleapis/gax-go/v2 v2.24.1/go.mod h1:bWeBei0NVwaNZKb2y1HUBS7gLXIF3/Tu3pq7j8D2Tb0=
github.com/googleapis/enterprise-certificate-proxy v0.3.24 h1:G9iCJIY+AYVmUskKdjX8FVy7sG+nbzjOQ41cvlmhcvg=
github.com/googleapis/enterprise-certificate-proxy v0.3.24/go.mod h1:hEjCmnoakQzv5ImLUp6alkY7XgLlNoAxwTXmhHHNIsI=
github.com/googleapis/gax-go/v2 v2.27.0 h1:qe2iQOUzxb3fd6x6ewpIEixPQhJ+0Ihc/RRZhKA/2g0=
github.com/googleapis/gax-go/v2 v2.27.0/go.mod h1:RofiqDw3ehcG4L0H1zO3D2Itpwd3/uS/7wdmbNDF1DE=
github.com/gopherjs/gopherjs v1.17.2 h1:fQnZVsXk8uxXIStYb0N4bGk7jeyTalG/wsZjQ25dO0g=
github.com/gopherjs/gopherjs v1.17.2/go.mod h1:pRRIvn/QzFLrKfvEz3qUuEhtE/zLCWfreZ6J5gM2i+k=
github.com/gorilla/mux v1.8.1 h1:TuBL49tXwgrFYWhqrNgrUNEY92u81SPhu7sTdzQEiWY=
@@ -827,8 +827,8 @@ github.com/lpar/calendar v0.2.0 h1:A1kxv6sbvBHFUkd2XotanIRqEXQGreQOeuGhkJqIaRA=
github.com/lpar/calendar v0.2.0/go.mod h1:fsVJa4o2NvXYzaCE5RXMadqbZzWEhNCGOUPVHTIZdjc=
github.com/lufia/plan9stats v0.0.0-20260627054121-477a66015f15 h1:YkjVPl/YH5XlJ+/NiwzJtPYXXKRcyjmEUhsDci6YK3c=
github.com/lufia/plan9stats v0.0.0-20260627054121-477a66015f15/go.mod h1:autxFIvghDt3jPTLoqZ9OZ7s9qTGNAWmYCjVFWPX/zg=
github.com/magiconair/properties v1.8.10 h1:s31yESBquKXCV9a/ScB3ESkOjUYYv+X0rg8SYxI99mE=
github.com/magiconair/properties v1.8.10/go.mod h1:Dhd985XPs7jluiymwWYZ0G4Z61jb3vdS329zhj2hYo0=
github.com/magiconair/properties v1.18.12 h1:sT9zQpvTB3B4gzrX0tmZNTEaGyg8Zw55MFYRE32Mr9I=
github.com/magiconair/properties v1.18.12/go.mod h1:Dhd985XPs7jluiymwWYZ0G4Z61jb3vdS329zhj2hYo0=
github.com/mailru/easyjson v0.7.7/go.mod h1:xzfreul335JAWq5oZzymOObrkdz5UnU4kGfJJLY9Nlc=
github.com/mailru/easyjson v0.9.2 h1:dX8U45hQsZpxd80nLvDGihsQ/OxlvTkVUXH2r/8cb2M=
github.com/mailru/easyjson v0.9.2/go.mod h1:1+xMtQp2MRNVL/V1bOzuP3aP8VNwRW55fQUto+XFtTU=
@@ -840,9 +840,8 @@ github.com/mattn/go-isatty v0.0.12/go.mod h1:cbi8OIDigv2wuxKPP5vlRcQ1OAZbq2CE4Ky
github.com/mattn/go-isatty v0.0.14/go.mod h1:7GGIvUiUoEMVVmxf/4nioHXj79iQHKdU27kJ6hsGG94=
github.com/mattn/go-isatty v0.0.24 h1:tGZZoVgT/KiqK1c8ocVLeDS8BSWMRd47J3Lbz7vsReI=
github.com/mattn/go-isatty v0.0.24/go.mod h1:nMCL3Zebbrt45jsMDgnfIwz6ydEQApk5oEI3HqDio6A=
github.com/mattn/go-runewidth v0.0.3/go.mod h1:LwmH8dsx7+W8Uxz3IHJYH5QSwggIsqBzpuz5H//U1FU=
github.com/mattn/go-runewidth v0.0.24 h1:cpokDiIn0MGnhdHwuWnJBITySJ20QyNGnY2kR/ay2DU=
github.com/mattn/go-runewidth v0.0.24/go.mod h1:XBkDxAl56ILZc9knddidhrOlY5R/pDhgLpndooCuJAs=
github.com/mattn/go-runewidth v0.0.31 h1:1oCV4ReTlosDBc6rYRw5cL8S/4y8iv5oRl4SJ5IBgwo=
github.com/mattn/go-runewidth v0.0.31/go.mod h1:3qAiGCV4Koz/yuveO58qUefmUTRm8r0IGEXZ9jeHp/8=
github.com/mattn/go-shellwords v1.0.13 h1:DC0OMEpGjm6LfNFU4ckYcvbQKyp2vE8atyFGXNtDcf4=
github.com/mattn/go-shellwords v1.0.13/go.mod h1:EZzvwXDESEeg03EKmM+RmDnNOPKG4lLtQsUlTZDWQ8Y=
github.com/mattn/go-sqlite3 v1.14.34 h1:3NtcvcUnFBPsuRcno8pUtupspG/GM+9nZ88zgJcp6Zk=
@@ -866,20 +865,22 @@ github.com/moby/buildkit v0.31.1 h1:j3p55abBl4kiXXPZgYX+6zWgB2aefqHXoPown12fIzU=
github.com/moby/buildkit v0.31.1/go.mod h1:YM5iNEbNCc6L1Zt3YWFB/aXNLufvf4Rcu0DPlc9HwQg=
github.com/moby/docker-image-spec v1.3.1 h1:jMKff3w6PgbfSa69GfNg+zN/XLhfXJGnEx3Nl2EsFP0=
github.com/moby/docker-image-spec v1.3.1/go.mod h1:eKmb5VW8vQEh/BAr2yvVNvuiJuY6UIocYsFu/DxxRpo=
github.com/moby/go-archive v0.3.0 h1:nos4BtzzUIqB406BgQnWGMI4qib9BZ8XUHU+ucv/n1c=
github.com/moby/go-archive v0.3.0/go.mod h1:Npdv43fFqlhZW7Xo8fbm3ZMYFvAGNviUPqX21VERbcE=
github.com/moby/go-archive v0.3.3 h1:OxxR9paxsluYi+zDUEXTTaIxtkK3viymW+Ka7vRhhME=
github.com/moby/go-archive v0.3.3/go.mod h1:Npdv43fFqlhZW7Xo8fbm3ZMYFvAGNviUPqX21VERbcE=
github.com/moby/locker v1.0.1 h1:fOXqR41zeveg4fFODix+1Ch4mj/gT0NE1XJbp/epuBg=
github.com/moby/locker v1.0.1/go.mod h1:S7SDdo5zpBK84bzzVlKr2V0hz+7x9hWbYC/kq7oQppc=
github.com/moby/moby/api v1.55.0 h1:2/sexvQyqIWS8pRSCFddBfpW2qE7vR7FCL+vN8pxwMc=
github.com/moby/moby/api v1.55.0/go.mod h1:+RQ6wluLwtYaTd1WnPLykIDPekkuyD/ROWQClE83pzs=
github.com/moby/moby/client v0.5.0 h1:5XhyPk2fuOWf6RlSFa3MkIIgDZkF25xToXW8Q/BH7cc=
github.com/moby/moby/client v0.5.0/go.mod h1:rcVpF8ncl9vo5gaIBdol6CnbEtSj1uxMvEV/UrykF/s=
github.com/moby/moby/api v1.56.1 h1:PpWkvVPB7Fr/No8w+TfyJ/I6rWZ2YPZhT3JorIg+06c=
github.com/moby/moby/api v1.56.1/go.mod h1:sZ+THbVWkjOmBPPfbnzdD/G1LuIexWhqlSHHPTDQ1Uk=
github.com/moby/moby/client v0.6.2 h1:ekSPlMccrZGcw6Scb8T9D39qE0mB+XupU/yS0zNw9N4=
github.com/moby/moby/client v0.6.2/go.mod h1:DGcnO7qmPJrqobx/ggQgQ/AUk+2wgVdQv7uxB0/mEsc=
github.com/moby/patternmatcher v0.6.1 h1:qlhtafmr6kgMIJjKJMDmMWq7WLkKIo23hsrpR3x084U=
github.com/moby/patternmatcher v0.6.1/go.mod h1:hDPoyOpDY7OrrMDLaYoY3hf52gNCR/YOUYxkhApJIxc=
github.com/moby/sys/atomicwriter v0.1.0 h1:kw5D/EqkBwsBFi0ss9v1VG3wIkVhzGvLklJ+w3A14Sw=
github.com/moby/sys/atomicwriter v0.1.0/go.mod h1:Ul8oqv2ZMNHOceF643P6FKPXeCmYtlQMvpizfsSoaWs=
github.com/moby/sys/capability v0.4.0 h1:4D4mI6KlNtWMCM1Z/K0i7RV1FkX+DBDHKVJpCndZoHk=
github.com/moby/sys/capability v0.4.0/go.mod h1:4g9IK291rVkms3LKCDOoYlnV8xKwoDTpIrNEE35Wq0I=
github.com/moby/sys/mount v0.3.5 h1:eS3fsZTjHaBihwjp4/+5Z3jxqLXYsbwxqpVSfFv3M00=
github.com/moby/sys/mount v0.3.5/go.mod h1:WUQDO+/uCiCIkIztx8SrwIDVn2dtMFRBebRhpDFT71M=
github.com/moby/sys/mountinfo v0.7.2 h1:1shs6aH5s4o5H2zQLn796ADW1wMrIwHsyJ2v9KouLrg=
github.com/moby/sys/mountinfo v0.7.2/go.mod h1:1YOa8w8Ih7uW0wALDUgT1dTTSBrZ+HiBLGws92L2RU4=
github.com/moby/sys/sequential v0.7.0 h1:ASQNGNROJSuOO6LL6bPHbKvuZu6NU8P4ldPWk31zj/8=
@@ -962,7 +963,6 @@ github.com/parquet-go/jsonlite v1.0.0 h1:87QNdi56wOfsE5bdgas0vRzHPxfJgzrXGml1zZd
github.com/parquet-go/jsonlite v1.0.0/go.mod h1:nDjpkpL4EOtqs6NQugUsi0Rleq9sW/OtC1NnZEnxzF0=
github.com/parquet-go/parquet-go v0.32.0 h1:NWDqTUHfrCS4cJP/Fj2HlxvqsrVedWG3sayMkf+znzM=
github.com/parquet-go/parquet-go v0.32.0/go.mod h1:navtkAYr2LGoJVp141oXPlO/sxLvaOe3la2JEoD8+rg=
github.com/pascaldekloe/goe v0.1.0 h1:cBOtyMzM9HTpWjXfbbunk26uA6nG3a8n06Wieeh0MwY=
github.com/pascaldekloe/goe v0.1.0/go.mod h1:lzWF7FIEvWOWxwDKqyGYQf6ZUaNfKdP144TG7ZOy1lc=
github.com/patrickmn/go-cache v2.1.0+incompatible h1:HRMgzkcYKYpi3C8ajMPV8OFXaaRUnok+kx1WdO15EQc=
github.com/patrickmn/go-cache v2.1.0+incompatible/go.mod h1:3Qf8kWWT7OJRJbdiICTKqZju1ZixQ/KpMGzzAfe6+WQ=
@@ -971,8 +971,8 @@ github.com/pelletier/go-toml/v2 v2.4.3 h1:GTRvJQutkOSftxIFD5xw9aepkYNuPWmVJpffdD
github.com/pelletier/go-toml/v2 v2.4.3/go.mod h1:2gIqNv+qfxSVS7cM2xJQKtLSTLUE9V8t9Stt+h56mCY=
github.com/pengsrc/go-shared v0.2.1-0.20190131101655-1999055a4a14 h1:XeOYlK9W1uCmhjJSsY78Mcuh7MVkNjTzmHx1yBzizSU=
github.com/pengsrc/go-shared v0.2.1-0.20190131101655-1999055a4a14/go.mod h1:jVblp62SafmidSkvWrXyxAme3gaTfEtWwRPGz5cpvHg=
github.com/peterh/liner v1.2.2 h1:aJ4AOodmL+JxOZZEL2u9iJf8omNRpqHc/EbrK+3mAXw=
github.com/peterh/liner v1.2.2/go.mod h1:xFwJyiKIXJZUKItq5dGHZSTBRAuG/CpeNpWLyiNRNwI=
github.com/peterh/liner v1.2.3 h1:JF4G03iaZjtPyJE7gdOk76/fvs4OObMJWQHwR3wphxE=
github.com/peterh/liner v1.2.3/go.mod h1:IC6gIBWLzuPGXmiX/86/ieO9K+9ITsaE6SKm1XkU9uw=
github.com/petermattis/goid v0.0.0-20180202154549-b0b1615b78e5/go.mod h1:jvVRKCrJTQWu0XVbaOlby/2lO20uSCHEMzzplHXte1o=
github.com/petermattis/goid v0.0.0-20260113132338-7c7de50cc741 h1:KPpdlQLZcHfTMQRi6bFQ7ogNO0ltFT4PmtwTLW4W+14=
github.com/petermattis/goid v0.0.0-20260113132338-7c7de50cc741/go.mod h1:pxMtw7cyUw6B2bRH0ZBANSPg+AoSud1I1iyJHI69jH4=
@@ -1016,8 +1016,8 @@ github.com/pmezard/go-difflib v1.0.1-0.20181226105442-5d4384ee4fb2 h1:Jamvg5psRI
github.com/pmezard/go-difflib v1.0.1-0.20181226105442-5d4384ee4fb2/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4=
github.com/posener/complete v1.2.3 h1:NP0eAhjcjImqslEwo/1hq7gpajME0fTLTezBKDqfXqo=
github.com/posener/complete v1.2.3/go.mod h1:WZIdtGGp+qx0sLrYKtIRAruyNpv6hFCicSgv7Sy7s/s=
github.com/power-devops/perfstat v0.0.0-20240221224432-82ca36839d55 h1:o4JXh1EVt9k/+g42oCprj/FisM4qX9L3sZB3upGN2ZU=
github.com/power-devops/perfstat v0.0.0-20240221224432-82ca36839d55/go.mod h1:OmDBASR4679mdNQnz2pUhc2G8CO2JrUAVFDRBDP/hJE=
github.com/power-devops/perfstat v0.0.0-20260805114148-88456608a4f6 h1:jL3a8soXdzuTCcRnKhOmtcsVOObdDTFf4O2B403HPRU=
github.com/power-devops/perfstat v0.0.0-20260805114148-88456608a4f6/go.mod h1:OmDBASR4679mdNQnz2pUhc2G8CO2JrUAVFDRBDP/hJE=
github.com/pquerna/cachecontrol v0.2.0 h1:vBXSNuE5MYP9IJ5kjsdo8uq+w41jSPgvba2DEnkRx9k=
github.com/pquerna/cachecontrol v0.2.0/go.mod h1:NrUG3Z7Rdu85UNR3vm7SOsl1nFIeSiQnrHV5K9mBcUI=
github.com/pquerna/ffjson v0.0.0-20190930134022-aa0246cd15f7/go.mod h1:YARuvh7BUWHNhzDq2OM5tzR2RiCcN2D7sapiKyCel/M=
@@ -1041,8 +1041,8 @@ github.com/prometheus/common v0.71.0/go.mod h1:CLJ5H8TEsGX8bl31BdMkfhIZ+QmZ9tBPP
github.com/prometheus/procfs v0.0.0-20181005140218-185b4288413d/go.mod h1:c3At6R/oaqEKCNdg8wHV1ftS6bRYblBhIjjI8uT2IGk=
github.com/prometheus/procfs v0.0.2/go.mod h1:TjEm7ze935MbeOT/UhFTIMYKhuLP4wbCsTZCD3I8kEA=
github.com/prometheus/procfs v0.0.8/go.mod h1:7Qr8sr6344vo1JqZ6HhLceV9o3AJ1Ff+GxbHq6oeK9A=
github.com/prometheus/procfs v0.22.0 h1:6q9+/JL9IKAPbCmBrv9n5O5Ty3NKnciV5X7YGw0oics=
github.com/prometheus/procfs v0.22.0/go.mod h1:CvmFr/GVhIjIvWJZW3tgkODBQMRIf0EyWMQLHCHab58=
github.com/prometheus/procfs v0.23.0 h1:ad+S6FTkzwaMgacKLM5h6bNKExS+DMH0YiUwXItH/o4=
github.com/prometheus/procfs v0.23.0/go.mod h1:3AVXO0FqLafyYFU/V/IQRZlC8Rgqz8Dm/x6pKLK+RIk=
github.com/putdotio/go-putio/putio v0.0.0-20200123120452-16d982cac2b8 h1:Y258uzXU/potCYnQd1r6wlAnoMB68BiCkCcCnKx1SH8=
github.com/putdotio/go-putio/putio v0.0.0-20200123120452-16d982cac2b8/go.mod h1:bSJjRokAHHOhA+XFxplld8w2R/dXLH7Z3BZ532vhFwU=
github.com/puzpuzpuz/xsync/v3 v3.5.1 h1:GJYJZwO6IdxN/IKbneznS6yPkVC+c3zyY/j19c++5Fg=
@@ -1106,8 +1106,8 @@ github.com/seaweedfs/go-fuse/v2 v2.9.4 h1:ACyloiuopdhRSjdLLeSWbsVaemMPskORaRF01T
github.com/seaweedfs/go-fuse/v2 v2.9.4/go.mod h1:zABdmWEa6A0bwaBeEOBUeUkGIZlxUhcdv+V1Dcc/U/I=
github.com/seaweedfs/goexif v1.0.3 h1:ve/OjI7dxPW8X9YQsv3JuVMaxEyF9Rvfd04ouL+Bz30=
github.com/seaweedfs/goexif v1.0.3/go.mod h1:Oni780Z236sXpIQzk1XoJlTwqrJ02smEin9zQeff7Fk=
github.com/seaweedfs/raft v1.2.3 h1:/NMrdhztjx8ojeu+aAPa2GmuVShQXp8YwmNmsMPU63E=
github.com/seaweedfs/raft v1.2.3/go.mod h1:fgs/rAVEzjQ7e04XMzG3eJhwZZRmBW+2uRtjakeCGeU=
github.com/seaweedfs/raft v1.2.4 h1:lEBQ9seDSGrJNs2WitMgfYpaDxLgh9qukm1IJTEV1hk=
github.com/seaweedfs/raft v1.2.4/go.mod h1:fgs/rAVEzjQ7e04XMzG3eJhwZZRmBW+2uRtjakeCGeU=
github.com/secure-systems-lab/go-securesystemslib v0.11.0 h1:iuCR9kcMFD4QurdKrGvPLoKZLv9YvwPYVr0473BdtFs=
github.com/secure-systems-lab/go-securesystemslib v0.11.0/go.mod h1:+PMOTjUGwHj2vcZ+TFKlb1tXRbrdWE1LYDT5i9JC80Q=
github.com/sergi/go-diff v1.0.0/go.mod h1:0CfEIISq7TuYL3j771MWULgwwjU+GofnZX9QAmXWZgo=
@@ -1116,8 +1116,8 @@ github.com/sergi/go-diff v1.2.0 h1:XU+rvMAioB0UC3q1MFrIQy4Vo5/4VsRDQQXHsEya6xQ=
github.com/sergi/go-diff v1.2.0/go.mod h1:STckp+ISIX8hZLjrqAeVduY0gWCT9IjLuqbuNXdaHfM=
github.com/shibumi/go-pathspec v1.3.0 h1:QUyMZhFo0Md5B8zV8x2tesohbb5kfbpTi9rBnKh5dkI=
github.com/shibumi/go-pathspec v1.3.0/go.mod h1:Xutfslp817l2I1cZvgcfeMQJG5QnU2lh5tVaaMCl3jE=
github.com/shirou/gopsutil/v4 v4.26.7 h1:IXzpHz/dkMRYAhKkOXr1HB6SuzWU3eoyyeWe7g3bNZc=
github.com/shirou/gopsutil/v4 v4.26.7/go.mod h1:5O9FjBiXoTDFatIWjZZosqj4pV0DRtLx598xGbBehzM=
github.com/shirou/gopsutil/v4 v4.26.10 h1:MJN6euesJx53MVEjscYeyRSRcrdIcy3sz5k9nosjldg=
github.com/shirou/gopsutil/v4 v4.26.10/go.mod h1:XpkIrcUMQhwte/dGxu57KtnMMFsbKelI7PP23zweKBk=
github.com/sigstore/sigstore v1.10.8 h1:1Mgkxvkw4AXMfIP1DOjc6kw0GkUgA8pGVpveN/EfOq4=
github.com/sigstore/sigstore v1.10.8/go.mod h1:f9+B/4iaYimvUkySyb2mvc73n3RLqNn24grHZM/ET8M=
github.com/sigstore/sigstore-go v1.2.1 h1:YWP/rDbBaEBvtbkj6xtwsSj38ZCFEhTVVadNOXjVe3A=
@@ -1200,8 +1200,8 @@ github.com/tarantool/go-option v1.1.0 h1:ShoOhNsdL41sRpm4hXCRDjV8H0WzPkd4UnKhLKb
github.com/tarantool/go-option v1.1.0/go.mod h1:hMr9z2JXOWlgdCBpCPSL2nwp8718GKYvNBJ+ZuzJbCo=
github.com/tarantool/go-tarantool/v3 v3.0.2 h1:9ZtHllun80QX7KS9tqd3dXa+QAu2BfqFtX1ZWcNnaW8=
github.com/tarantool/go-tarantool/v3 v3.0.2/go.mod h1:TXxLWhUCgdxXFfelnTSkq+goKRTRj660zxq4/WXPe8k=
github.com/testcontainers/testcontainers-go v0.44.0 h1:/Fwh6HY1mIikhnm9e7HwoxGycx0lzRAE0f5VQpjFxzI=
github.com/testcontainers/testcontainers-go v0.44.0/go.mod h1:IcnwQrYTO86xHXu5bvMaBH7ATlbS3Qn1M1QWW3c66rE=
github.com/testcontainers/testcontainers-go v0.45.0 h1:ss9Stm9KDX6JLgGL4jr2fC5dwpabTbCBsRcGadBPR9k=
github.com/testcontainers/testcontainers-go v0.45.0/go.mod h1:VMJZ7M+qhgfEHWNbtjm596OyEOraULSTDaus/h0hmJI=
github.com/testcontainers/testcontainers-go/modules/compose v0.44.0 h1:8YcW51jhgpkkiRVe10Wj9TCBthJmoNpU2fK5WSf7TQ8=
github.com/testcontainers/testcontainers-go/modules/compose v0.44.0/go.mod h1:0du5YaH5n8Of4bDM/RwIt421Q14kvqrXZ0mB8EjuMoo=
github.com/the42/cartconvert v0.0.0-20131203171324-aae784c392b8 h1:I4DY8wLxJXCrMYzDM6lKCGc3IQwJX0PlTLsd3nQqI3c=
@@ -1353,8 +1353,8 @@ go.etcd.io/bbolt v1.5.0 h1:S7GAl7Fxv12yohbwFfIbQCGDWbQbtDGPET4P/bD4lxU=
go.etcd.io/bbolt v1.5.0/go.mod h1:mkltfYE5aUHQxUct9N9V+Kp7aSjFqjgrhcXIS70Lrdk=
go.etcd.io/etcd/api/v3 v3.7.1 h1:KJG0/DcWGfe3Y1otDf/fsBf0TSSgpxZ5RO/L8SFt73E=
go.etcd.io/etcd/api/v3 v3.7.1/go.mod h1:8bXIpCMeV7E3/XL0Ix123ATn3dB+0V7d9zklHbB0m78=
go.etcd.io/etcd/client/pkg/v3 v3.7.1 h1:rKYsj3pRkR0eK3yjT3XOgrhqfmIfj9pzNgxjh7mfFv4=
go.etcd.io/etcd/client/pkg/v3 v3.7.1/go.mod h1:cnzZGIUzSfjEwLC6UBVsSXlEK1eepS/JUD7wE6PLRT0=
go.etcd.io/etcd/client/pkg/v3 v3.7.3 h1:5zMwNu3ycrzCeA5NwOfQbxFRhqq974chq1DrYMua4QI=
go.etcd.io/etcd/client/pkg/v3 v3.7.3/go.mod h1:/OhzqYrF0U7d1NOk55VMyUQGbYD1WoIe4bJwxb47wOk=
go.etcd.io/etcd/client/v3 v3.7.1 h1:0PEMMC0KuZmVIN+RAbdqfkZ45pYTgKVtmBEbRCvZFUg=
go.etcd.io/etcd/client/v3 v3.7.1/go.mod h1:ffNqALa8tRCYhYo1F9oR489y23K39Gz+BSR3ApAGYq0=
go.mongodb.org/mongo-driver v1.17.9 h1:IexDdCuuNJ3BHrELgBlyaH9p60JXAvdzWR128q+U5tU=
@@ -1371,8 +1371,8 @@ go.opentelemetry.io/contrib/instrumentation/net/http/httptrace/otelhttptrace v0.
go.opentelemetry.io/contrib/instrumentation/net/http/httptrace/otelhttptrace v0.69.0/go.mod h1:3jnStNwSufK+f5ktjL4EPcwtig4rtd81NS70lqHuXl8=
go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp v0.71.0 h1:3g7B90UzBltIDKq1/5mrTGxTnOFDV0ICOhLoxiZ8jlg=
go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp v0.71.0/go.mod h1:Ef8SuTh59BT7+ofpDxN9z+yOlc4t2GjLmKDgYNJL/NU=
go.opentelemetry.io/otel v1.46.0 h1:FHt5/CDyVxi/8IM1CH7VE/rRgq3kLHa2mSTVMO8AWyc=
go.opentelemetry.io/otel v1.46.0/go.mod h1:Gj3SEScelsNC45tp4nSxRYlS+f5iez7W8XPMCt905kE=
go.opentelemetry.io/otel v1.47.0 h1:j7ALJ/zgkS7Z6aeJW09p8VC9804bC+PpeTfCD4XPnOM=
go.opentelemetry.io/otel v1.47.0/go.mod h1:8wS9O2qfXrYrzp6hIF/HOYJJf/wIhFPhR2xLuP+iXQU=
go.opentelemetry.io/otel/exporters/otlp/otlpmetric/otlpmetricgrpc v1.46.0 h1:qkDYCAFiZXLcs1L4aY+tP2wguQ4kURANqHOQMA2et2s=
go.opentelemetry.io/otel/exporters/otlp/otlpmetric/otlpmetricgrpc v1.46.0/go.mod h1:tkipS4DRzmpAmvg+Gw4++O1IdDq6TVDnvnYU6cmbQVs=
go.opentelemetry.io/otel/exporters/otlp/otlpmetric/otlpmetrichttp v1.44.0 h1:RuynHbfU8JUEw7DyONgkVYg2SVtsoF28y0LGIr69jgA=
@@ -1387,16 +1387,18 @@ go.opentelemetry.io/otel/exporters/stdout/stdoutmetric v1.46.0 h1:PR9eAf7o0dQs3h
go.opentelemetry.io/otel/exporters/stdout/stdoutmetric v1.46.0/go.mod h1:2Z4KyNdH1uuzivdinyfGsxzNNT/Rl45pwtVwfYVI0xk=
go.opentelemetry.io/otel/exporters/zipkin v1.45.0 h1:KN3btaILMTxR4QDHVGAO87lq5ButzK7l+kIfLuxQ1oA=
go.opentelemetry.io/otel/exporters/zipkin v1.45.0/go.mod h1:yNcodmUclM4InyWoOwX/YW4Jri0Gj5FWAlM+NqCrtqY=
go.opentelemetry.io/otel/metric v1.46.0 h1:yBnkXvgV7AXFILZc5K6IZe/CBFF3OS7BJ8ov6/lj0K8=
go.opentelemetry.io/otel/metric v1.46.0/go.mod h1:iPmdWqifKUdzziPkvvzIJXITl56fQx2mGM/DHLB3/2o=
go.opentelemetry.io/otel/metric/x v0.68.0 h1:TA/cBT23D3MnxYPwHL7YFOdYGdx0A0v+s7Mzotpd1dU=
go.opentelemetry.io/otel/metric/x v0.68.0/go.mod h1:agudOmvWhwUTjgibWDzxD2PoWYnpw5Ht5jISYOD2Hd4=
go.opentelemetry.io/otel/sdk v1.46.0 h1:h5CNQQjEbuQXY/JfZtgt3i7HVFV3aHPO2OAwO2eTYPI=
go.opentelemetry.io/otel/sdk v1.46.0/go.mod h1:GAERFXFt5SYCEB+YiKUbMBeza6UaDH7GmGOZEfh2gSM=
go.opentelemetry.io/otel/sdk/metric v1.46.0 h1:0piZ26EG4RBfebb2jhDH6ERCYHoVWduc3kLgPCwSnSE=
go.opentelemetry.io/otel/sdk/metric v1.46.0/go.mod h1:I1PbKrdVc8Qu8HYVDNtqVIwLwjNrhsV/uFuxfwg8mO4=
go.opentelemetry.io/otel/trace v1.46.0 h1:OULy7ccdJnZtJ0UDYFOIGaCmiWzJ8Vi2G/Rsu60qs1c=
go.opentelemetry.io/otel/trace v1.46.0/go.mod h1:J7GAXweO77XSFkB/rmAqk9D6ihszhFjLU+d9WuUxDLI=
go.opentelemetry.io/otel/log v1.47.0 h1:cOTS1CcLbSQeZKanGJ+0JpF/+t4PELi3O3bbl2lqCcI=
go.opentelemetry.io/otel/log v1.47.0/go.mod h1:9byitSQ5pLC6PpqwGXjqdMKya6ZTswHRZh2vvXT33nw=
go.opentelemetry.io/otel/metric v1.47.0 h1:4PptaldXx3Eat1XjMZ68pPJEs5wrhlemctZE9a3UdWY=
go.opentelemetry.io/otel/metric v1.47.0/go.mod h1:ADGSXxRrXM6bjbvLo535EstVFlPpPYZm4LBKixjDHwU=
go.opentelemetry.io/otel/metric/x v0.69.0 h1:DjRLr15H83v+hCW7JA9NoJvOkYTtmq5YoDRbe9deYpM=
go.opentelemetry.io/otel/metric/x v0.69.0/go.mod h1:uVvsMPMFFyj/HUQfrUnH3JjnOQ1dwFDorgFLRBasM0k=
go.opentelemetry.io/otel/sdk v1.47.0 h1:zWXEr4j2lFefG87TU6Yg8a7ngfohIKFZHKp0Hf5hC6I=
go.opentelemetry.io/otel/sdk v1.47.0/go.mod h1:VUc24kiOeoGsxG8G9ULx3fWKvB7jMhnGE8Oi607lgR0=
go.opentelemetry.io/otel/sdk/metric v1.47.0 h1:lfISg2j93VT6yqdk9OfUaZmw/GfcZqCCV3jdXtsPnKw=
go.opentelemetry.io/otel/sdk/metric v1.47.0/go.mod h1:ypLp+mW1Nt2x+Szt3b5/i1syodyts49lMOwxpDI3VGw=
go.opentelemetry.io/otel/trace v1.47.0 h1:JOjX/Oci8K94QHddo+bbfya/Ai/nf6/dt9ZfrFNWSrM=
go.opentelemetry.io/otel/trace v1.47.0/go.mod h1:jNaSLa2PZEYFG6fRjJABAu+bw4FS08uDmPg28lTghu0=
go.opentelemetry.io/proto/otlp v0.7.0/go.mod h1:PqfVotwruBrMGOCsRd/89rSnXhoiJIqeYNgFYFoEGnI=
go.opentelemetry.io/proto/otlp v1.11.0 h1:5rrYs0Ykyj50sdU/JU0x8etU+LubXWb+gED6TbEdMIk=
go.opentelemetry.io/proto/otlp v1.11.0/go.mod h1:SmVizdCOAm3XBtG1g1NnOdhW6jtddT72hLMhv8VwA8E=
@@ -1421,8 +1423,8 @@ go.uber.org/zap v1.28.0 h1:IZzaP1Fv73/T/pBMLk4VutPl36uNC+OSUh3JLG3FIjo=
go.uber.org/zap v1.28.0/go.mod h1:rDLpOi171uODNm/mxFcuYWxDsqWSAVkFdX4XojSKg/Q=
go.yaml.in/yaml/v2 v2.4.4 h1:tuyd0P+2Ont/d6e2rl3be67goVK4R6deVxCUX5vyPaQ=
go.yaml.in/yaml/v2 v2.4.4/go.mod h1:gMZqIpDtDqOfM0uNfy0SkpRhvUryYH0Z6wdMYcacYXQ=
go.yaml.in/yaml/v3 v3.0.5 h1:N6y/pJk8buWs9NY5ERU2HSMfm+IuD/OtfdAnq6kESPw=
go.yaml.in/yaml/v3 v3.0.5/go.mod h1:HVTZu1O7/Vkt2N+BFy8Zza+lnLsABggaTM2ZpNIGuKg=
go.yaml.in/yaml/v3 v3.0.6 h1:zcDfvjXEqTQmpjniUBLc91xYJMB2c4QOFxEF/LCEckM=
go.yaml.in/yaml/v3 v3.0.6/go.mod h1:HVTZu1O7/Vkt2N+BFy8Zza+lnLsABggaTM2ZpNIGuKg=
go.yaml.in/yaml/v4 v4.0.0-rc.6 h1:1h7H1ohdUh93/FyE4YaDa1Zh64K6VVbjF4K6WUxMtH4=
go.yaml.in/yaml/v4 v4.0.0-rc.6/go.mod h1:aZqd9kCMsGL7AuUv/m/PvWLdg5sjJsZ4oHDEnfPPfY0=
gocloud.dev v0.47.0 h1:qN4/0vtutn97f1X5ziisf3WXuaFL9AR0FIRnyJRYzgA=
@@ -1468,8 +1470,8 @@ golang.org/x/mod v0.7.0/go.mod h1:iBbtSCu2XBx23ZKBPSOrRkjjQPZFPuis4dIYUhu/chs=
golang.org/x/mod v0.8.0/go.mod h1:iBbtSCu2XBx23ZKBPSOrRkjjQPZFPuis4dIYUhu/chs=
golang.org/x/mod v0.12.0/go.mod h1:iBbtSCu2XBx23ZKBPSOrRkjjQPZFPuis4dIYUhu/chs=
golang.org/x/mod v0.13.0/go.mod h1:hTbmBsO62+eylJbnUtE2MGJUyE7QWk4xUqPFrRgJ+7c=
golang.org/x/mod v0.41.0 h1:qJmnOUb4YB+FsEuM3HcWucdZASCPGhsX6uljO6pog0c=
golang.org/x/mod v0.41.0/go.mod h1:Ek9pY8RKWXwsWvd3rQiHYtMqkjSUV+s1Rj7j4H5Ur6o=
golang.org/x/mod v0.42.0 h1:ICeL3t2L30ljSPHHxDnR4fqVcBt/zR2zJAb9juavmQg=
golang.org/x/mod v0.42.0/go.mod h1:u5fzIrRInkL8MaO1cbRkUHT64v303YIT5KEqpNr7jRs=
golang.org/x/net v0.0.0-20180724234803-3673e40ba225/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4=
golang.org/x/net v0.0.0-20180826012351-8a410e7b638d/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4=
golang.org/x/net v0.0.0-20180906233101-161cd47e91fd/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4=
@@ -1497,8 +1499,8 @@ golang.org/x/net v0.8.0/go.mod h1:QVkue5JL9kW//ek3r6jTKnTFis1tRmNAW2P1shuFdJc=
golang.org/x/net v0.10.0/go.mod h1:0qNGK6F8kojg2nk9dLZ2mShWaEBan6FAoqfSigmmuDg=
golang.org/x/net v0.15.0/go.mod h1:idbUs1IY1+zTqbi8yxTbhexhEEk5ur9LInksu6HrEpk=
golang.org/x/net v0.16.0/go.mod h1:NxSsAGuq816PNPmqtQdLE42eU2Fs7NoRIZrHJAlaCOE=
golang.org/x/net v0.61.0 h1:RR+6j/BTBCrggOLugxa+66R8FvoN6MHlzDlZIbN9YFw=
golang.org/x/net v0.61.0/go.mod h1:WgDCOTH8iwtB66T4/GfI94AHo/hhdb7Hah9NDseQfiI=
golang.org/x/net v0.62.0 h1:18ELalfM5+XNRwzM2nYqXsZQ9ByS1uhj2O/U7+5jqRY=
golang.org/x/net v0.62.0/go.mod h1:WgDCOTH8iwtB66T4/GfI94AHo/hhdb7Hah9NDseQfiI=
golang.org/x/oauth2 v0.0.0-20180821212333-d2e6202438be/go.mod h1:N/0e6XlmueqKjAGxoOufVs8QHGRruUQn6yWY3a++T0U=
golang.org/x/oauth2 v0.0.0-20200107190931-bf48bf16ab8d/go.mod h1:gOpvHmFTYa4IltrdGE7lF6nIHvwfUNPOp7c8zoXwtLw=
golang.org/x/oauth2 v0.37.0 h1:JUlcxA8oAtauLfiH8FX2/FkAWHAdi0QtGCGc+hofE98=
@@ -1540,12 +1542,10 @@ golang.org/x/sys v0.0.0-20210423082822-04245dca01da/go.mod h1:h1NjWce9XRLGQEsW7w
golang.org/x/sys v0.0.0-20210510120138-977fb7262007/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.0.0-20210514084401-e8d321eab015/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.0.0-20210615035016-665e8c7367d1/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.0.0-20210616094352-59db8d763f22/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.0.0-20210630005230-0f9fa26af87c/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.0.0-20210927094055-39ccf1dd6fa6/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.0.0-20211007075335-d3039528d8ac/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.0.0-20211025201205-69cdffdb9359/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.0.0-20211117180635-dee7805ff2e1/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.0.0-20220408201424-a24fb2fb8a0f/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.0.0-20220503163025-988cb79eb6c6/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.0.0-20220520151302-bc2c85ada10a/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
@@ -1620,8 +1620,8 @@ golang.org/x/xerrors v0.0.0-20240903120638-7835f813f4da h1:noIWHXmPHxILtqtCOPIhS
golang.org/x/xerrors v0.0.0-20240903120638-7835f813f4da/go.mod h1:NDW/Ps6MPRej6fsCIbMTohpP40sJ/P/vI1MoTEGwX90=
gonum.org/v1/gonum v0.17.0 h1:VbpOemQlsSMrYmn7T2OUvQ4dqxQXU+ouZFQsZOx50z4=
gonum.org/v1/gonum v0.17.0/go.mod h1:El3tOrEuMpv2UdMrbNlKEh9vd86bmQ6vqIcDwxEOc1E=
google.golang.org/api v0.297.0 h1:WktxTsnnx0yZNnsR6j0q6hR21RnnK81FHTOPy/ux4OE=
google.golang.org/api v0.297.0/go.mod h1:S4m8x0M6OkQpkOzGk1y9JG2sm4fFQrMh6dxzjCTszhE=
google.golang.org/api v0.310.0 h1:537ccDDmbwuwX1iof3Vtdz9vow6uJCgrSZJWedAWM0I=
google.golang.org/api v0.310.0/go.mod h1:5NUthU+LmGMKzYgRaScb+TTe1Lo7x4A+J7g+Yj8BRzA=
google.golang.org/appengine v1.1.0/go.mod h1:EbEs0AVv82hx2wNQdGPgUI5lhzA/G0D9YwlJXL52JkM=
google.golang.org/appengine v1.4.0/go.mod h1:xpcJRLb0r/rnEns0DIKYYv+WjYCduHsrkT7/EB5XEv4=
google.golang.org/appengine v1.6.7/go.mod h1:8WjMMxjGQR8xUklV/ARdw2HLXBOI7O7uCIDZVag1xfc=
@@ -1631,12 +1631,12 @@ google.golang.org/genproto v0.0.0-20200423170343-7949de9c1215/go.mod h1:55QSHmfG
google.golang.org/genproto v0.0.0-20200513103714-09dca8ec2884/go.mod h1:55QSHmfGQM9UVYDPBsyGGes0y52j32PQ3BqQfXhyH3c=
google.golang.org/genproto v0.0.0-20200526211855-cb27e3aa2013/go.mod h1:NbSheEEYHJ7i3ixzK3sjbqSGDJWnxyFXZblF3eUsNvo=
google.golang.org/genproto v0.0.0-20210903162649-d08c68adba83/go.mod h1:eFjDcFEctNawg4eG61bRv87N7iHBWyVhJu7u1kqDUXY=
google.golang.org/genproto v0.0.0-20260904194346-d0f1323225a4 h1:6QRNo34FxJcE1LG6XfnmaQ17m6w21oXaKlKoBaF+UqE=
google.golang.org/genproto v0.0.0-20260904194346-d0f1323225a4/go.mod h1:eifJ91nefXnfP/bMkZsACVGm14ZtEL4eUXRLTryfbY4=
google.golang.org/genproto/googleapis/api v0.0.0-20260921155816-b14227669459 h1:GS9OIt/j7c8bvBjYNgnKQysVfmV7e4jM0H8ZK95G4t8=
google.golang.org/genproto/googleapis/api v0.0.0-20260921155816-b14227669459/go.mod h1:PX5/4vemwVoXtwEcRDWwcR1/r0qrosfx3qoVADMwnVE=
google.golang.org/genproto/googleapis/rpc v0.0.0-20260921155816-b14227669459 h1:b0xCahf3FK2m2Cv0p4vTozGPWncCvLfwV86UNg8xWU8=
google.golang.org/genproto/googleapis/rpc v0.0.0-20260921155816-b14227669459/go.mod h1:OaIUM3+LpYcK2GXM4FTmhWoIq371Owdr+Cc7/BsYHHc=
google.golang.org/genproto v0.0.0-20261012220534-4341edf3f941 h1:CXq2XEi5aEyLjjPZIwqrmQH8ZY4ctUwK7uyAx3mjX/U=
google.golang.org/genproto v0.0.0-20261012220534-4341edf3f941/go.mod h1:3gK0ehnhP0V7/x3ypGoLyT3d0n6TSi/EcsphIjHrcz4=
google.golang.org/genproto/googleapis/api v0.0.0-20261012220534-4341edf3f941 h1:DS/w3N8rsfrwxwgGSj6WhMsPo7aNQGmDx74J81zd0NQ=
google.golang.org/genproto/googleapis/api v0.0.0-20261012220534-4341edf3f941/go.mod h1:AuGAdi9KnXss7xbLxIKhCvk+Ro/SxpET/iS45DNYN+M=
google.golang.org/genproto/googleapis/rpc v0.0.0-20261019180522-ce908d29299b h1:6WNWHZk91dp4gZ25642OALkuG1UVNZbwENJD19DnB8k=
google.golang.org/genproto/googleapis/rpc v0.0.0-20261019180522-ce908d29299b/go.mod h1:OaIUM3+LpYcK2GXM4FTmhWoIq371Owdr+Cc7/BsYHHc=
google.golang.org/grpc v1.19.0/go.mod h1:mqu4LbDTu4XGKhr4mRzUsmM4RtVoemTSY81AxZiDr8c=
google.golang.org/grpc v1.23.0/go.mod h1:Y5yQAOtifL1yxbo5wqy6BxZv8vAUGQwXBOALyacEbxg=
google.golang.org/grpc v1.25.1/go.mod h1:c3i+UQWmh7LiEpx4sFZnkU36qjEYZ0imhYfXVyQciAY=
+14 -18
View File
@@ -157,12 +157,20 @@ worker_asset_name() {
echo "weed-worker_${os}_${arch}.tar.gz"
}
# The vector engine behind the S3 Vectors API, built like the worker.
# The vector engine behind the S3 Vectors API, published for linux only like
# the worker.
vector_asset_name() {
local os="$1" arch="$2"
echo "weed-vector_${os}_${arch}.tar.gz"
}
vector_platform_supported() {
case "$1/$2" in
linux/amd64|linux/arm64) return 0 ;;
*) return 1 ;;
esac
}
# Install a single component
install_component() {
local component="$1" os="$2" arch="$3"
@@ -272,15 +280,7 @@ install_component() {
;;
vector)
# Published for linux amd64/arm64 only, like the worker.
case "$os" in
linux) ;;
*) error "The vector engine is not available for ${os}. Supported: linux" ;;
esac
case "$arch" in
amd64|arm64) ;;
*) error "The vector engine is not available for ${arch}. Supported: amd64, arm64" ;;
esac
vector_platform_supported "$os" "$arch" || error "The vector engine is not available for ${os}/${arch}. Supported: linux/amd64, linux/arm64"
asset_name="$(vector_asset_name "$os" "$arch")"
download_url="https://github.com/${REPO}/releases/download/${VERSION}/${asset_name}"
@@ -288,15 +288,11 @@ install_component() {
info "Extracting ${asset_name}..."
tar xzf "${tmpdir}/${asset_name}" -C "$tmpdir"
local vector_bin
vector_bin="$(find "$tmpdir" -name 'weed-vector' -type f | head -1)"
if [ -z "$vector_bin" ]; then
error "Could not find weed-vector binary in archive"
fi
vector_bin="$(find "$tmpdir" -name weed-vector -type f | head -1)"
[ -n "$vector_bin" ] || error "Could not find weed-vector binary in archive"
chmod +x "$vector_bin"
install_binary "$vector_bin" "weed-vector"
install_binary "$vector_bin" weed-vector
VECTOR_INSTALLED=true
ok "Installed weed-vector to ${INSTALL_DIR}/weed-vector"
;;
@@ -364,7 +360,7 @@ main() {
else
install_component "worker-rust" "$os" "$arch"
fi
if [ "$os" != "linux" ] || { [ "$arch" != "amd64" ] && [ "$arch" != "arm64" ]; }; then
if ! vector_platform_supported "$os" "$arch"; then
warn "Skipping the vector engine: no build for ${os}/${arch}"
elif ! asset_exists "https://github.com/${REPO}/releases/download/${VERSION}/$(vector_asset_name "$os" "$arch")"; then
warn "Skipping the vector engine: ${VERSION} does not carry one"
+2 -2
View File
@@ -1,6 +1,6 @@
apiVersion: v1
description: SeaweedFS
name: seaweedfs
appVersion: "4.52"
appVersion: "4.53"
# Dev note: Trigger a helm chart release by `git tag -a helm-<version>`
version: 4.52.0
version: 4.53.1
+44
View File
@@ -687,3 +687,47 @@ generates a new UUID and the license stops matching. See
[Master data](#master-data-hostpath-vs-a-claim). Check the binding with
`kubectl exec <master-pod> -- curl -s localhost:9333/license/status`
(`cluster_uuid` must equal `license_uuid`).
## Vector buckets
The S3 gateway serves the S3 Vectors API on its existing endpoint. Enable a vector engine and Lance maintenance worker alongside the cluster:
```yaml
s3:
enabled: true
vector:
enabled: true
replicas: 1
worker:
enabled: true
admin:
enabled: true
secret:
adminPassword: change-me
global:
seaweedfs:
securityConfig:
jwtSigning:
filerWrite: true
```
The engine registers its pod address with the masters and stores indexes through the filers and volume servers. Both Go and Rust volume servers use the same setup. The maintenance worker builds and updates ANN indexes; queries can scan data before indexing completes. Enabling filer write signing also requires clients writing directly to the filer HTTP API to supply a signed JWT.
The bundled engine images support Linux amd64 and arm64. Use `vector.imageOverride` to select a compatible engine build, `vector.masters` for an external cluster, and `vector.resources` to set memory and CPU limits. The cache defaults to 512 MiB and is configured with `vector.readCacheMB`.
`global.seaweedfs.enableSecurity` provisions and mounts the vector certificate and shared security configuration. With external certificates, provide the `<fullname>-vector-cert` secret (`my-seaweedfs-vector-cert` for release `my`). `fullname` follows `seaweedfs.fullname`, including `fullnameOverride`, `nameOverride`, and DNS-name truncation. Provide this secret alongside the existing CA and client secrets.
The default `vector.metricsPort: 0` and TCP probes support existing engine images. The default 4.53 image does not accept metrics flags; enabling metrics with it is rejected at render time. For engine builds with `/metrics`, `/health`, and `/ready`, set `vector.imageOverride` to that compatible image and `vector.metricsPort: 9329`; readiness then checks master registration. Enabling `global.seaweedfs.monitoring.enabled` also creates the vector ServiceMonitor. NetworkPolicy includes the configured gRPC and metrics ports.
To provision vector buckets during installation and upgrades, set:
```yaml
vector:
enabled: true
createBuckets:
- embeddings
```
The hook uses the AWS CLI's S3 Vectors API and works with existing SeaweedFS vector releases. It creates missing buckets, preserves existing vector buckets, and fails if the name belongs to an Iceberg bucket. It uses the chart's generated S3 admin credentials, including `s3.credentials.admin.existingSecret`. With a custom `s3.existingConfigSecret` or an external endpoint, set `vector.bucketHook.credentialsSecret` to a Secret containing `AWS_ACCESS_KEY_ID` and `AWS_SECRET_ACCESS_KEY` (and optionally `AWS_SESSION_TOKEN`). Key names are configurable. `vector.bucketHook.endpoint` overrides the internal S3 service URL; `caBundleSecret` and `caBundleKey` configure a private HTTPS CA. The hook image and resource limits are configurable under `vector.bucketHook`. For restricted Pod Security, select a non-root-compatible image and configure `vector.bucketHook.podSecurityContext` and `containerSecurityContext`, including `runAsNonRoot` and the image user ID.
@@ -3951,6 +3951,970 @@
],
"title": "Filer Object Size Distribution",
"type": "bargauge"
},
{
"type": "row",
"title": "S3 Vectors",
"collapsed": true,
"id": 95,
"gridPos": {
"h": 1,
"w": 24,
"x": 0,
"y": 139
},
"panels": [
{
"datasource": {
"type": "prometheus",
"uid": "${DS_PROMETHEUS}"
},
"description": "Vector engine metrics require HTTP metrics to be enabled. Gateway bucket labels require vector bucket attribution.",
"fieldConfig": {
"defaults": {
"color": {
"mode": "palette-classic"
},
"custom": {
"axisBorderShow": false,
"axisCenteredZero": false,
"axisColorMode": "text",
"axisLabel": "",
"axisPlacement": "auto",
"barAlignment": 0,
"drawStyle": "line",
"fillOpacity": 0,
"gradientMode": "none",
"hideFrom": {
"legend": false,
"tooltip": false,
"viz": false
},
"insertNulls": false,
"lineInterpolation": "linear",
"lineWidth": 1,
"pointSize": 4,
"scaleDistribution": {
"type": "linear"
},
"showPoints": "auto",
"spanNulls": false,
"stacking": {
"group": "A",
"mode": "none"
},
"thresholdsStyle": {
"mode": "off"
}
},
"mappings": [],
"thresholds": {
"mode": "absolute",
"steps": [
{
"color": "green",
"value": null
}
]
},
"unit": "reqps",
"unitScale": true
},
"overrides": []
},
"gridPos": {
"h": 8,
"w": 12,
"x": 0,
"y": 140
},
"id": 96,
"links": [],
"options": {
"legend": {
"calcs": [],
"displayMode": "list",
"placement": "bottom",
"showLegend": true
},
"tooltip": {
"mode": "single",
"sort": "none"
}
},
"pluginVersion": "8.1.2",
"targets": [
{
"datasource": {
"type": "prometheus",
"uid": "${DS_PROMETHEUS}"
},
"editorMode": "code",
"expr": "sum by (type, bucket) (rate(SeaweedFS_s3_request_total{namespace=~\"$NAMESPACE\",type=~\"s3vectors\\\\..*\"}[$__rate_interval]))",
"range": true,
"refId": "A",
"legendFormat": "{{type}} {{bucket}}"
}
],
"title": "Vector API requests by bucket",
"type": "timeseries"
},
{
"datasource": {
"type": "prometheus",
"uid": "${DS_PROMETHEUS}"
},
"description": "Vector engine metrics require HTTP metrics to be enabled. Gateway bucket labels require vector bucket attribution.",
"fieldConfig": {
"defaults": {
"color": {
"mode": "palette-classic"
},
"custom": {
"axisBorderShow": false,
"axisCenteredZero": false,
"axisColorMode": "text",
"axisLabel": "",
"axisPlacement": "auto",
"barAlignment": 0,
"drawStyle": "line",
"fillOpacity": 0,
"gradientMode": "none",
"hideFrom": {
"legend": false,
"tooltip": false,
"viz": false
},
"insertNulls": false,
"lineInterpolation": "linear",
"lineWidth": 1,
"pointSize": 4,
"scaleDistribution": {
"type": "linear"
},
"showPoints": "auto",
"spanNulls": false,
"stacking": {
"group": "A",
"mode": "none"
},
"thresholdsStyle": {
"mode": "off"
}
},
"mappings": [],
"thresholds": {
"mode": "absolute",
"steps": [
{
"color": "green",
"value": null
}
]
},
"unit": "s",
"unitScale": true
},
"overrides": []
},
"gridPos": {
"h": 8,
"w": 12,
"x": 12,
"y": 140
},
"id": 97,
"links": [],
"options": {
"legend": {
"calcs": [],
"displayMode": "list",
"placement": "bottom",
"showLegend": true
},
"tooltip": {
"mode": "single",
"sort": "none"
}
},
"pluginVersion": "8.1.2",
"targets": [
{
"datasource": {
"type": "prometheus",
"uid": "${DS_PROMETHEUS}"
},
"editorMode": "code",
"expr": "histogram_quantile(0.95, sum by (le, bucket) (rate(SeaweedFS_s3_request_seconds_bucket{namespace=~\"$NAMESPACE\",type=\"s3vectors.QueryVectors\"}[$__rate_interval])))",
"range": true,
"refId": "A",
"legendFormat": "{{bucket}}"
}
],
"title": "Vector query p95 latency",
"type": "timeseries"
},
{
"datasource": {
"type": "prometheus",
"uid": "${DS_PROMETHEUS}"
},
"description": "Vector engine metrics require HTTP metrics to be enabled. Gateway bucket labels require vector bucket attribution.",
"fieldConfig": {
"defaults": {
"color": {
"mode": "palette-classic"
},
"custom": {
"axisBorderShow": false,
"axisCenteredZero": false,
"axisColorMode": "text",
"axisLabel": "",
"axisPlacement": "auto",
"barAlignment": 0,
"drawStyle": "line",
"fillOpacity": 0,
"gradientMode": "none",
"hideFrom": {
"legend": false,
"tooltip": false,
"viz": false
},
"insertNulls": false,
"lineInterpolation": "linear",
"lineWidth": 1,
"pointSize": 4,
"scaleDistribution": {
"type": "linear"
},
"showPoints": "auto",
"spanNulls": false,
"stacking": {
"group": "A",
"mode": "none"
},
"thresholdsStyle": {
"mode": "off"
}
},
"mappings": [],
"thresholds": {
"mode": "absolute",
"steps": [
{
"color": "green",
"value": null
}
]
},
"unit": "reqps",
"unitScale": true
},
"overrides": []
},
"gridPos": {
"h": 8,
"w": 12,
"x": 0,
"y": 148
},
"id": 98,
"links": [],
"options": {
"legend": {
"calcs": [],
"displayMode": "list",
"placement": "bottom",
"showLegend": true
},
"tooltip": {
"mode": "single",
"sort": "none"
}
},
"pluginVersion": "8.1.2",
"targets": [
{
"datasource": {
"type": "prometheus",
"uid": "${DS_PROMETHEUS}"
},
"editorMode": "code",
"expr": "sum by (type, code) (rate(SeaweedFS_s3_request_total{namespace=~\"$NAMESPACE\",type=~\"s3vectors\\\\..*\",code=~\"[45]..\"}[$__rate_interval]))",
"range": true,
"refId": "A",
"legendFormat": "{{type}} {{code}}"
}
],
"title": "Vector API errors",
"type": "timeseries"
},
{
"datasource": {
"type": "prometheus",
"uid": "${DS_PROMETHEUS}"
},
"description": "Vector engine metrics require HTTP metrics to be enabled. Gateway bucket labels require vector bucket attribution.",
"fieldConfig": {
"defaults": {
"color": {
"mode": "palette-classic"
},
"custom": {
"axisBorderShow": false,
"axisCenteredZero": false,
"axisColorMode": "text",
"axisLabel": "",
"axisPlacement": "auto",
"barAlignment": 0,
"drawStyle": "line",
"fillOpacity": 0,
"gradientMode": "none",
"hideFrom": {
"legend": false,
"tooltip": false,
"viz": false
},
"insertNulls": false,
"lineInterpolation": "linear",
"lineWidth": 1,
"pointSize": 4,
"scaleDistribution": {
"type": "linear"
},
"showPoints": "auto",
"spanNulls": false,
"stacking": {
"group": "A",
"mode": "none"
},
"thresholdsStyle": {
"mode": "off"
}
},
"mappings": [],
"thresholds": {
"mode": "absolute",
"steps": [
{
"color": "green",
"value": null
}
]
},
"unit": "short",
"unitScale": true
},
"overrides": []
},
"gridPos": {
"h": 8,
"w": 12,
"x": 12,
"y": 148
},
"id": 99,
"links": [],
"options": {
"legend": {
"calcs": [],
"displayMode": "list",
"placement": "bottom",
"showLegend": true
},
"tooltip": {
"mode": "single",
"sort": "none"
}
},
"pluginVersion": "8.1.2",
"targets": [
{
"datasource": {
"type": "prometheus",
"uid": "${DS_PROMETHEUS}"
},
"editorMode": "code",
"expr": "last_over_time(SeaweedFS_vector_connected{namespace=~\"$NAMESPACE\"}[5m]) * on (cluster, namespace, instance, job) up{namespace=~\"$NAMESPACE\"}",
"range": true,
"refId": "A",
"legendFormat": "{{instance}}"
}
],
"title": "Registered vector engines",
"type": "timeseries"
},
{
"datasource": {
"type": "prometheus",
"uid": "${DS_PROMETHEUS}"
},
"description": "Vector engine metrics require HTTP metrics to be enabled. Gateway bucket labels require vector bucket attribution.",
"fieldConfig": {
"defaults": {
"color": {
"mode": "palette-classic"
},
"custom": {
"axisBorderShow": false,
"axisCenteredZero": false,
"axisColorMode": "text",
"axisLabel": "",
"axisPlacement": "auto",
"barAlignment": 0,
"drawStyle": "line",
"fillOpacity": 0,
"gradientMode": "none",
"hideFrom": {
"legend": false,
"tooltip": false,
"viz": false
},
"insertNulls": false,
"lineInterpolation": "linear",
"lineWidth": 1,
"pointSize": 4,
"scaleDistribution": {
"type": "linear"
},
"showPoints": "auto",
"spanNulls": false,
"stacking": {
"group": "A",
"mode": "none"
},
"thresholdsStyle": {
"mode": "off"
}
},
"mappings": [],
"thresholds": {
"mode": "absolute",
"steps": [
{
"color": "green",
"value": null
}
]
},
"unit": "reqps",
"unitScale": true
},
"overrides": []
},
"gridPos": {
"h": 8,
"w": 12,
"x": 0,
"y": 156
},
"id": 100,
"links": [],
"options": {
"legend": {
"calcs": [],
"displayMode": "list",
"placement": "bottom",
"showLegend": true
},
"tooltip": {
"mode": "single",
"sort": "none"
}
},
"pluginVersion": "8.1.2",
"targets": [
{
"datasource": {
"type": "prometheus",
"uid": "${DS_PROMETHEUS}"
},
"editorMode": "code",
"expr": "sum by (type, code) (rate(SeaweedFS_vector_request_total{namespace=~\"$NAMESPACE\",code!=\"OK\"}[$__rate_interval]))",
"range": true,
"refId": "A",
"legendFormat": "{{type}} {{code}}"
}
],
"title": "Vector engine request failures",
"type": "timeseries"
},
{
"datasource": {
"type": "prometheus",
"uid": "${DS_PROMETHEUS}"
},
"description": "Vector engine metrics require HTTP metrics to be enabled. Gateway bucket labels require vector bucket attribution.",
"fieldConfig": {
"defaults": {
"color": {
"mode": "palette-classic"
},
"custom": {
"axisBorderShow": false,
"axisCenteredZero": false,
"axisColorMode": "text",
"axisLabel": "",
"axisPlacement": "auto",
"barAlignment": 0,
"drawStyle": "line",
"fillOpacity": 0,
"gradientMode": "none",
"hideFrom": {
"legend": false,
"tooltip": false,
"viz": false
},
"insertNulls": false,
"lineInterpolation": "linear",
"lineWidth": 1,
"pointSize": 4,
"scaleDistribution": {
"type": "linear"
},
"showPoints": "auto",
"spanNulls": false,
"stacking": {
"group": "A",
"mode": "none"
},
"thresholdsStyle": {
"mode": "off"
}
},
"mappings": [],
"thresholds": {
"mode": "absolute",
"steps": [
{
"color": "green",
"value": null
}
]
},
"unit": "short",
"unitScale": true
},
"overrides": []
},
"gridPos": {
"h": 8,
"w": 12,
"x": 12,
"y": 156
},
"id": 101,
"links": [],
"options": {
"legend": {
"calcs": [],
"displayMode": "list",
"placement": "bottom",
"showLegend": true
},
"tooltip": {
"mode": "single",
"sort": "none"
}
},
"pluginVersion": "8.1.2",
"targets": [
{
"datasource": {
"type": "prometheus",
"uid": "${DS_PROMETHEUS}"
},
"editorMode": "code",
"expr": "sum by (type) (SeaweedFS_vector_in_flight_requests{namespace=~\"$NAMESPACE\"})",
"range": true,
"refId": "A",
"legendFormat": "{{type}}"
}
],
"title": "Vector engine requests in flight",
"type": "timeseries"
},
{
"datasource": {
"type": "prometheus",
"uid": "${DS_PROMETHEUS}"
},
"description": "Vector engine metrics require HTTP metrics to be enabled. Gateway bucket labels require vector bucket attribution.",
"fieldConfig": {
"defaults": {
"color": {
"mode": "palette-classic"
},
"custom": {
"axisBorderShow": false,
"axisCenteredZero": false,
"axisColorMode": "text",
"axisLabel": "",
"axisPlacement": "auto",
"barAlignment": 0,
"drawStyle": "line",
"fillOpacity": 0,
"gradientMode": "none",
"hideFrom": {
"legend": false,
"tooltip": false,
"viz": false
},
"insertNulls": false,
"lineInterpolation": "linear",
"lineWidth": 1,
"pointSize": 4,
"scaleDistribution": {
"type": "linear"
},
"showPoints": "auto",
"spanNulls": false,
"stacking": {
"group": "A",
"mode": "none"
},
"thresholdsStyle": {
"mode": "off"
}
},
"mappings": [],
"thresholds": {
"mode": "absolute",
"steps": [
{
"color": "green",
"value": null
}
]
},
"unit": "percentunit",
"unitScale": true
},
"overrides": []
},
"gridPos": {
"h": 8,
"w": 12,
"x": 0,
"y": 164
},
"id": 102,
"links": [],
"options": {
"legend": {
"calcs": [],
"displayMode": "list",
"placement": "bottom",
"showLegend": true
},
"tooltip": {
"mode": "single",
"sort": "none"
}
},
"pluginVersion": "8.1.2",
"targets": [
{
"datasource": {
"type": "prometheus",
"uid": "${DS_PROMETHEUS}"
},
"editorMode": "code",
"expr": "(sum by (cluster, namespace, job, instance) (rate(SeaweedFS_vector_cache_requests_total{namespace=~\"$NAMESPACE\",result=\"hit\"}[$__rate_interval])) or on (cluster, namespace, job, instance) (0 * sum by (cluster, namespace, job, instance) (rate(SeaweedFS_vector_cache_requests_total{namespace=~\"$NAMESPACE\"}[$__rate_interval])))) / sum by (cluster, namespace, job, instance) (rate(SeaweedFS_vector_cache_requests_total{namespace=~\"$NAMESPACE\"}[$__rate_interval]))",
"range": true,
"refId": "A",
"legendFormat": "{{cluster}} {{namespace}} {{instance}}"
}
],
"title": "Vector cache hit ratio",
"type": "timeseries"
},
{
"datasource": {
"type": "prometheus",
"uid": "${DS_PROMETHEUS}"
},
"description": "Vector engine metrics require HTTP metrics to be enabled. Gateway bucket labels require vector bucket attribution.",
"fieldConfig": {
"defaults": {
"color": {
"mode": "palette-classic"
},
"custom": {
"axisBorderShow": false,
"axisCenteredZero": false,
"axisColorMode": "text",
"axisLabel": "",
"axisPlacement": "auto",
"barAlignment": 0,
"drawStyle": "line",
"fillOpacity": 0,
"gradientMode": "none",
"hideFrom": {
"legend": false,
"tooltip": false,
"viz": false
},
"insertNulls": false,
"lineInterpolation": "linear",
"lineWidth": 1,
"pointSize": 4,
"scaleDistribution": {
"type": "linear"
},
"showPoints": "auto",
"spanNulls": false,
"stacking": {
"group": "A",
"mode": "none"
},
"thresholdsStyle": {
"mode": "off"
}
},
"mappings": [],
"thresholds": {
"mode": "absolute",
"steps": [
{
"color": "green",
"value": null
}
]
},
"unit": "bytes",
"unitScale": true
},
"overrides": []
},
"gridPos": {
"h": 8,
"w": 12,
"x": 12,
"y": 164
},
"id": 103,
"links": [],
"options": {
"legend": {
"calcs": [],
"displayMode": "list",
"placement": "bottom",
"showLegend": true
},
"tooltip": {
"mode": "single",
"sort": "none"
}
},
"pluginVersion": "8.1.2",
"targets": [
{
"datasource": {
"type": "prometheus",
"uid": "${DS_PROMETHEUS}"
},
"editorMode": "code",
"expr": "SeaweedFS_vector_cache_bytes{namespace=~\"$NAMESPACE\"}",
"range": true,
"refId": "A",
"legendFormat": "{{instance}}"
}
],
"title": "Vector cache memory",
"type": "timeseries"
},
{
"datasource": {
"type": "prometheus",
"uid": "${DS_PROMETHEUS}"
},
"description": "Vector engine metrics require HTTP metrics to be enabled. Gateway bucket labels require vector bucket attribution.",
"fieldConfig": {
"defaults": {
"color": {
"mode": "palette-classic"
},
"custom": {
"axisBorderShow": false,
"axisCenteredZero": false,
"axisColorMode": "text",
"axisLabel": "",
"axisPlacement": "auto",
"barAlignment": 0,
"drawStyle": "line",
"fillOpacity": 0,
"gradientMode": "none",
"hideFrom": {
"legend": false,
"tooltip": false,
"viz": false
},
"insertNulls": false,
"lineInterpolation": "linear",
"lineWidth": 1,
"pointSize": 4,
"scaleDistribution": {
"type": "linear"
},
"showPoints": "auto",
"spanNulls": false,
"stacking": {
"group": "A",
"mode": "none"
},
"thresholdsStyle": {
"mode": "off"
}
},
"mappings": [],
"thresholds": {
"mode": "absolute",
"steps": [
{
"color": "green",
"value": null
}
]
},
"unit": "Bps",
"unitScale": true
},
"overrides": []
},
"gridPos": {
"h": 8,
"w": 12,
"x": 0,
"y": 172
},
"id": 104,
"links": [],
"options": {
"legend": {
"calcs": [],
"displayMode": "list",
"placement": "bottom",
"showLegend": true
},
"tooltip": {
"mode": "single",
"sort": "none"
}
},
"pluginVersion": "8.1.2",
"targets": [
{
"datasource": {
"type": "prometheus",
"uid": "${DS_PROMETHEUS}"
},
"editorMode": "code",
"expr": "rate(SeaweedFS_vector_storage_read_bytes_total{namespace=~\"$NAMESPACE\"}[$__rate_interval])",
"range": true,
"refId": "A",
"legendFormat": "{{instance}}"
}
],
"title": "Vector storage read bandwidth",
"type": "timeseries"
},
{
"datasource": {
"type": "prometheus",
"uid": "${DS_PROMETHEUS}"
},
"description": "Vector engine metrics require HTTP metrics to be enabled. Gateway bucket labels require vector bucket attribution.",
"fieldConfig": {
"defaults": {
"color": {
"mode": "palette-classic"
},
"custom": {
"axisBorderShow": false,
"axisCenteredZero": false,
"axisColorMode": "text",
"axisLabel": "",
"axisPlacement": "auto",
"barAlignment": 0,
"drawStyle": "line",
"fillOpacity": 0,
"gradientMode": "none",
"hideFrom": {
"legend": false,
"tooltip": false,
"viz": false
},
"insertNulls": false,
"lineInterpolation": "linear",
"lineWidth": 1,
"pointSize": 4,
"scaleDistribution": {
"type": "linear"
},
"showPoints": "auto",
"spanNulls": false,
"stacking": {
"group": "A",
"mode": "none"
},
"thresholdsStyle": {
"mode": "off"
}
},
"mappings": [],
"thresholds": {
"mode": "absolute",
"steps": [
{
"color": "green",
"value": null
}
]
},
"unit": "ops",
"unitScale": true
},
"overrides": []
},
"gridPos": {
"h": 8,
"w": 12,
"x": 12,
"y": 172
},
"id": 105,
"links": [],
"options": {
"legend": {
"calcs": [],
"displayMode": "list",
"placement": "bottom",
"showLegend": true
},
"tooltip": {
"mode": "single",
"sort": "none"
}
},
"pluginVersion": "8.1.2",
"targets": [
{
"datasource": {
"type": "prometheus",
"uid": "${DS_PROMETHEUS}"
},
"editorMode": "code",
"expr": "sum by (instance) (rate(SeaweedFS_worker_lance_vector_indexes_built_total{namespace=~\"$NAMESPACE\"}[$__rate_interval]))",
"range": true,
"refId": "A",
"legendFormat": "{{instance}}"
}
],
"title": "Vector index maintenance builds",
"type": "timeseries"
}
]
}
],
"refresh": "",
@@ -4039,6 +5003,6 @@
"timezone": "browser",
"title": "SeaweedFS",
"uid": "a24009d7-cbda-4443-a132-1cc1c4677304",
"version": 1,
"version": 2,
"weekStart": ""
}
@@ -0,0 +1,50 @@
{{- include "seaweedfs.compat" . -}}
{{- if and .Values.vector.enabled .Values.global.seaweedfs.enableSecurity (not .Values.certificates.externalCertificates.enabled)}}
apiVersion: cert-manager.io/v1{{ if .Values.global.seaweedfs.certificates.alphacrds }}alpha1{{ end }}
kind: Certificate
metadata:
name: {{ include "seaweedfs.fullname" . }}-vector-cert
namespace: {{ include "seaweedfs.namespace" . | quote }}
labels:
app.kubernetes.io/name: {{ template "seaweedfs.name" . }}
helm.sh/chart: {{ .Chart.Name }}-{{ .Chart.Version | replace "+" "_" }}
app.kubernetes.io/managed-by: {{ .Release.Service }}
app.kubernetes.io/instance: {{ .Release.Name | quote }}
app.kubernetes.io/component: vector
{{- with include "seaweedfs.commonLabels" $ }}{{ . | nindent 4 }}{{ end }}
{{- if .Values.vector.annotations }}
annotations:
{{- toYaml .Values.vector.annotations | nindent 4 }}
{{- end }}
spec:
secretName: {{ include "seaweedfs.fullname" . }}-vector-cert
issuerRef:
name: {{ include "seaweedfs.fullname" . }}-ca-issuer
kind: Issuer
commonName: {{ .Values.certificates.commonName }}
subject:
organizations:
- "SeaweedFS CA"
dnsNames:
- '*.{{ include "seaweedfs.fullname" . }}-vector'
- '*.{{ include "seaweedfs.fullname" . }}-vector.{{ include "seaweedfs.namespace" . }}'
- '*.{{ include "seaweedfs.fullname" . }}-vector.{{ include "seaweedfs.namespace" . }}.svc'
- '*.{{ include "seaweedfs.fullname" . }}-vector.{{ include "seaweedfs.namespace" . }}.svc.cluster.local'
{{- if .Values.certificates.dnsNames }}
{{- range .Values.certificates.dnsNames }}
- {{ . | quote }}
{{- end }}
{{- end }}
{{- if .Values.certificates.ipAddresses }}
ipAddresses:
{{- range .Values.certificates.ipAddresses }}
- {{ . }}
{{- end }}
{{- end }}
privateKey:
algorithm: {{ .Values.certificates.keyAlgorithm }}
size: {{ .Values.certificates.keySize }}
duration: {{ .Values.certificates.duration }}
renewBefore: {{ .Values.certificates.renewBefore }}
{{- end }}
@@ -770,3 +770,15 @@ on install.
{{- $context := merge (dict "Values" (omit .Values "commonLabels")) (omit . "Values") -}}
{{- include (print .Template.BasePath "/master/master-configmap.yaml") $context | sha256sum -}}
{{- end -}}
{{- define "seaweedfs.vector.selectorLabels" -}}
app.kubernetes.io/name: {{ include "seaweedfs.name" . }}
app.kubernetes.io/instance: {{ .Release.Name | quote }}
app.kubernetes.io/component: vector
{{- end -}}
{{- define "seaweedfs.vector.labels" -}}
{{ include "seaweedfs.vector.selectorLabels" . }}
app.kubernetes.io/managed-by: {{ .Release.Service }}
helm.sh/chart: {{ .Chart.Name }}-{{ .Chart.Version | replace "+" "_" }}
{{- with include "seaweedfs.commonLabels" . }}{{ . | nindent 0 }}{{ end }}
{{- end -}}
@@ -110,6 +110,18 @@
{{- $targets = append $targets (dict "component" "worker" "ports" $ports) }}
{{- end }}
{{- if .Values.vector.enabled }}
{{- $ports := list .Values.vector.port }}
{{- if .Values.vector.metricsPort }}
{{- $ports = append $ports .Values.vector.metricsPort }}
{{- end }}
{{- $targets = append $targets (dict "component" "vector" "ports" $ports) }}
{{- end }}
{{- if and .Values.vector.enabled .Values.vector.createBuckets }}
{{- $targets = append $targets (dict "component" "vector-bucket-hook" "ports" (list)) }}
{{- end }}
{{- if .Values.cosi.enabled }}
{{- /* The provisioner sidecar watches the COSI CRs, so it needs the API
server; nothing needs to reach the driver itself. */}}
@@ -143,7 +155,7 @@
every component the chart can produce rather than the ones enabled here,
so one values file can still carry overrides for a component this release
leaves off. */}}
{{- $known := list "master" "filer" "s3" "sftp" "admin" "worker" "volume" "bucket-hook" "volume-resize-hook" "objectstorage-provisioner" "seaweedfs-all-in-one" }}
{{- $known := list "vector" "vector-bucket-hook" "master" "filer" "s3" "sftp" "admin" "worker" "volume" "bucket-hook" "volume-resize-hook" "objectstorage-provisioner" "seaweedfs-all-in-one" }}
{{- range $vname, $_ := (.Values.volumes | default dict) }}
{{- $known = append $known (trimSuffix "-" (printf "volume-%s" $vname)) }}
{{- end }}
@@ -215,8 +215,11 @@ spec:
{{- end }}
{{- end }}
{{- range $createBuckets }}
{{- if .ttl }}
/bin/echo 'fs.configure -locationPrefix={{ $bucketsFolder }}/{{ .name }}/ -ttl={{ .ttl }} -apply' | /usr/bin/weed shell
{{- if not (or (kindIs "invalid" .replication) (and (kindIs "string" .replication) (or (eq .replication "") (regexMatch "^([01][0-9]{2}|2[0-4][0-9]|25[0-5])$" .replication)))) }}
{{- fail (printf "createBuckets %s: replication must be a quoted three-digit string from 000 to 255 such as \"001\" (use --set-string)" .name) }}
{{- end }}
{{- if or .ttl .replication .disk }}
/bin/echo 'fs.configure -locationPrefix={{ $bucketsFolder }}/{{ .name }}/{{- with .ttl }} -ttl={{ . }}{{- end }}{{- with .replication }} -replication={{ . }}{{- end }}{{- with .disk }} -disk={{ . }}{{- end }} -apply' | /usr/bin/weed shell
{{- end }}
{{- end }}
{{- range $createBuckets }}
@@ -94,6 +94,10 @@ data:
cert = "/usr/local/share/ca-certificates/admin/tls.crt"
key = "/usr/local/share/ca-certificates/admin/tls.key"
[grpc.vector]
cert = "/usr/local/share/ca-certificates/vector/tls.crt"
key = "/usr/local/share/ca-certificates/vector/tls.key"
[grpc.worker]
cert = "/usr/local/share/ca-certificates/worker/tls.crt"
key = "/usr/local/share/ca-certificates/worker/tls.key"
@@ -0,0 +1,167 @@
{{- include "seaweedfs.compat" . -}}
{{- if and .Values.vector.enabled .Values.vector.createBuckets }}
{{- $hook := .Values.vector.bucketHook }}
{{- $component := "s3" }}
{{- $port := .Values.s3.port }}
{{- $enabled := .Values.s3.enabled }}
{{- $auth := .Values.s3.enableAuth }}
{{- $configSecret := .Values.s3.existingConfigSecret }}
{{- if and .Values.allInOne.enabled .Values.allInOne.s3.enabled }}
{{- $component = "all-in-one" }}
{{- $port = .Values.allInOne.s3.port | default .Values.s3.port }}
{{- $enabled = true }}
{{- $auth = or .Values.allInOne.s3.enableAuth .Values.s3.enableAuth .Values.filer.s3.enableAuth }}
{{- $configSecret = or .Values.allInOne.s3.existingConfigSecret .Values.s3.existingConfigSecret .Values.filer.s3.existingConfigSecret }}
{{- else if and (not $enabled) .Values.filer.s3.enabled }}
{{- $component = "filer" }}
{{- $port = .Values.filer.s3.port }}
{{- $enabled = true }}
{{- $auth = .Values.filer.s3.enableAuth }}
{{- $configSecret = .Values.filer.s3.existingConfigSecret }}
{{- end }}
{{- if and (not $enabled) (not $hook.endpoint) }}
{{- fail "vector.createBuckets requires an enabled S3 gateway or vector.bucketHook.endpoint" }}
{{- end }}
{{- range .Values.vector.createBuckets }}
{{- if not (kindIs "string" .) }}{{ fail "vector.createBuckets entries must be bucket names" }}{{ end }}
{{- if not (and (ge (len .) 3) (le (len .) 63) (regexMatch "^[a-z0-9][a-z0-9-]*[a-z0-9]$" .)) }}
{{- fail "vector.createBuckets names must be 3-63 lowercase letters, digits or hyphens" }}
{{- end }}
{{- end }}
{{- $credentials := .Values.s3.credentials | default dict }}
{{- $admin := $credentials.admin | default dict }}
{{- $secret := $hook.credentialsSecret }}
{{- $accessKey := $hook.accessKeyKey }}
{{- $secretKey := $hook.secretKeyKey }}
{{- if and $auth (not $secret) }}
{{- if $configSecret }}{{ fail "vector.bucketHook.credentialsSecret is required with s3.existingConfigSecret" }}{{ end }}
{{- $secret = $admin.existingSecret | default (printf "%s-s3-secret" (include "seaweedfs.fullname" .)) }}
{{- $accessKey = $admin.accessKeyKey | default "admin_access_key_id" }}
{{- $secretKey = $admin.secretKeyKey | default "admin_secret_access_key" }}
{{- end }}
apiVersion: batch/v1
kind: Job
metadata:
name: {{ include "seaweedfs.fullname" . }}-vector-bucket-hook
namespace: {{ include "seaweedfs.namespace" . | quote }}
labels:
app.kubernetes.io/name: {{ include "seaweedfs.name" . }}
app.kubernetes.io/instance: {{ .Release.Name | quote }}
app.kubernetes.io/component: vector-bucket-hook
{{- with include "seaweedfs.commonLabels" . }}{{ . | nindent 4 }}{{ end }}
annotations:
helm.sh/hook: post-install,post-upgrade
helm.sh/hook-weight: "-5"
helm.sh/hook-delete-policy: before-hook-creation,hook-succeeded
spec:
backoffLimit: 2
activeDeadlineSeconds: 360
template:
metadata:
labels:
app.kubernetes.io/name: {{ include "seaweedfs.name" . }}
app.kubernetes.io/instance: {{ .Release.Name | quote }}
app.kubernetes.io/component: vector-bucket-hook
spec:
restartPolicy: Never
automountServiceAccountToken: false
{{- with $hook.podSecurityContext }}
securityContext:
{{- toYaml . | nindent 8 }}
{{- end }}
{{- with .Values.vector.nodeSelector }}
nodeSelector:
{{- toYaml . | nindent 8 }}
{{- end }}
{{- with .Values.vector.tolerations }}
tolerations:
{{- toYaml . | nindent 8 }}
{{- end }}
{{- include "seaweedfs.imagePullSecrets" . | nindent 6 }}
containers:
- name: create-vector-buckets
image: {{ $hook.image | quote }}
command: ["/bin/sh", "-ec"]
args:
- |
for bucket in {{ range .Values.vector.createBuckets }}{{ . | quote }} {{ end }}; do
attempt=0
while ! aws --endpoint-url "$S3_ENDPOINT" s3vectors get-vector-bucket --vector-bucket-name "$bucket" >/dev/null 2>/tmp/bucket-error; do
if grep -q NotFound /tmp/bucket-error; then
aws --endpoint-url "$S3_ENDPOINT" s3vectors create-vector-bucket --vector-bucket-name "$bucket" ||
aws --endpoint-url "$S3_ENDPOINT" s3vectors get-vector-bucket --vector-bucket-name "$bucket"
break
fi
if grep -q AccessDenied /tmp/bucket-error; then cat /tmp/bucket-error >&2; exit 1; fi
attempt=$((attempt + 1))
if [ "$attempt" -ge 60 ]; then cat /tmp/bucket-error >&2; exit 1; fi
sleep 5
done
done
env:
- name: S3_ENDPOINT
value: {{ $hook.endpoint | default (printf "http://%s.%s:%d" (include "seaweedfs.componentName" (list . $component)) (include "seaweedfs.namespace" .) (int $port)) | quote }}
- name: AWS_DEFAULT_REGION
value: {{ $hook.region | quote }}
- name: AWS_EC2_METADATA_DISABLED
value: "true"
- name: AWS_PAGER
value: ""
- name: AWS_MAX_ATTEMPTS
value: "2"
- name: AWS_ACCESS_KEY_ID
{{- if $secret }}
valueFrom:
secretKeyRef:
name: {{ $secret | quote }}
key: {{ $accessKey | quote }}
{{- else }}
value: seaweedfs
{{- end }}
- name: AWS_SECRET_ACCESS_KEY
{{- if $secret }}
valueFrom:
secretKeyRef:
name: {{ $secret | quote }}
key: {{ $secretKey | quote }}
{{- else }}
value: seaweedfs
{{- end }}
{{- if $hook.credentialsSecret }}
- name: AWS_SESSION_TOKEN
valueFrom:
secretKeyRef:
name: {{ $secret | quote }}
key: {{ $hook.sessionTokenKey | quote }}
optional: true
{{- end }}
{{- if $hook.caBundleSecret }}
- name: AWS_CA_BUNDLE
value: /etc/ssl/vector/ca.crt
{{- end }}
securityContext:
{{- toYaml $hook.containerSecurityContext | nindent 12 }}
{{- with $hook.resources }}
resources:
{{- toYaml . | nindent 12 }}
{{- end }}
volumeMounts:
- name: tmp
mountPath: /tmp
{{- if $hook.caBundleSecret }}
- name: ca
mountPath: /etc/ssl/vector
readOnly: true
{{- end }}
volumes:
- name: tmp
emptyDir: {}
{{- if $hook.caBundleSecret }}
- name: ca
secret:
secretName: {{ $hook.caBundleSecret | quote }}
items:
- key: {{ $hook.caBundleKey | quote }}
path: ca.crt
{{- end }}
{{- end }}
@@ -0,0 +1,166 @@
{{- include "seaweedfs.compat" . -}}
{{- if .Values.vector.enabled }}
{{- if and .Values.vector.metricsPort (not .Values.vector.imageOverride) (eq (toString (.Values.image.tag | default .Chart.AppVersion)) "4.53") }}
{{- fail "vector.metricsPort requires vector.imageOverride selecting an engine with metrics support; the default 4.53 image does not support metrics" }}
{{- end }}
apiVersion: apps/v1
kind: Deployment
metadata:
name: {{ include "seaweedfs.componentName" (list . "vector") }}
namespace: {{ include "seaweedfs.namespace" . | quote }}
labels:
{{- include "seaweedfs.vector.labels" . | nindent 4 }}
{{- with .Values.vector.annotations }}
annotations:
{{- toYaml . | nindent 4 }}
{{- end }}
spec:
replicas: {{ .Values.vector.replicas }}
selector:
matchLabels:
{{- include "seaweedfs.vector.selectorLabels" . | nindent 6 }}
template:
metadata:
labels:
{{- include "seaweedfs.vector.selectorLabels" . | nindent 8 }}
{{- with .Values.podLabels }}{{ toYaml . | nindent 8 }}{{ end }}
{{- with .Values.vector.podLabels }}{{ toYaml . | nindent 8 }}{{ end }}
annotations:
{{- with .Values.podAnnotations }}{{ toYaml . | nindent 8 }}{{ end }}
{{- with .Values.vector.podAnnotations }}{{ toYaml . | nindent 8 }}{{ end }}
spec:
{{- include "seaweedfs.imagePullSecrets" . | nindent 6 }}
automountServiceAccountToken: {{ .Values.global.seaweedfs.automountServiceAccountToken }}
{{- with .Values.vector.nodeSelector }}
nodeSelector:
{{- toYaml . | nindent 8 }}
{{- end }}
{{- with .Values.vector.affinity }}
affinity:
{{- toYaml . | nindent 8 }}
{{- end }}
{{- with .Values.vector.tolerations }}
tolerations:
{{- toYaml . | nindent 8 }}
{{- end }}
{{- if and .Values.vector.podSecurityContext (or (not (hasKey .Values.vector.podSecurityContext "enabled")) .Values.vector.podSecurityContext.enabled) }}
securityContext:
{{- omit .Values.vector.podSecurityContext "enabled" | toYaml | nindent 8 }}
{{- end }}
containers:
- name: vector
image: {{ .Values.vector.imageOverride | default (include "seaweedfs.image" .) | quote }}
imagePullPolicy: {{ .Values.global.seaweedfs.imagePullPolicy }}
{{- $masters := include "seaweedfs.masterServerArg" . }}
{{- if .Values.allInOne.enabled }}{{ $masters = include "seaweedfs.cluster.masterAddress" . }}{{ end }}
command: ["/bin/sh", "-ec"]
args:
- 'case "$POD_IP" in *:*) POD_IP="[$POD_IP]";; esac; exec /usr/bin/weed-vector --listen="$POD_IP:{{ .Values.vector.port }}" "$@"'
- vector
- {{ printf "--masters=%s" (.Values.vector.masters | default $masters) | quote }}
- {{ printf "--read-cache-mb=%d" (int .Values.vector.readCacheMB) | quote }}
{{- if .Values.filer.filerGroup }}
- {{ printf "--filer-group=%s" .Values.filer.filerGroup | quote }}
{{- end }}
{{- if .Values.vector.nprobes }}
- {{ printf "--nprobes=%d" (int .Values.vector.nprobes) | quote }}
{{- end }}
{{- if include "seaweedfs.securityConfigEnabled" . }}
- --security-toml=/etc/seaweedfs/security.toml
{{- end }}
{{- if .Values.vector.metricsPort }}
- {{ printf "--metrics-port=%d" (int .Values.vector.metricsPort) | quote }}
- --metrics-ip=$(POD_IP)
{{- end }}
{{- range .Values.vector.extraArgs }}
- {{ . | quote }}
{{- end }}
env:
- name: POD_IP
valueFrom:
fieldRef:
fieldPath: status.podIP
{{- $env := dict }}
{{- include "seaweedfs.mergeExtraEnvironmentVars" (dict "global" .Values.global.seaweedfs "component" .Values.vector "target" $env) }}
{{- range $key := keys $env | sortAlpha }}
- name: {{ $key }}
{{- $value := index $env $key }}
{{- if kindIs "map" $value }}
valueFrom:
{{- toYaml $value | nindent 16 }}
{{- else }}
value: {{ tpl (toString $value) $ | quote }}
{{- end }}
{{- end }}
ports:
- name: grpc
containerPort: {{ .Values.vector.port }}
{{- if .Values.vector.metricsPort }}
- name: metrics
containerPort: {{ .Values.vector.metricsPort }}
{{- end }}
startupProbe:
tcpSocket:
port: grpc
periodSeconds: 5
failureThreshold: 60
readinessProbe:
{{- if .Values.vector.metricsPort }}
httpGet:
path: /ready
port: metrics
{{- else }}
tcpSocket:
port: grpc
{{- end }}
periodSeconds: 5
livenessProbe:
{{- if .Values.vector.metricsPort }}
httpGet:
path: /health
port: metrics
{{- else }}
tcpSocket:
port: grpc
{{- end }}
initialDelaySeconds: 10
periodSeconds: 10
{{- if or (not (hasKey .Values.vector.containerSecurityContext "enabled")) .Values.vector.containerSecurityContext.enabled }}
securityContext:
{{- omit .Values.vector.containerSecurityContext "enabled" | toYaml | nindent 12 }}
{{- end }}
{{- with .Values.vector.resources }}
resources:
{{- toYaml . | nindent 12 }}
{{- end }}
volumeMounts:
- name: tmp
mountPath: /tmp
{{- if include "seaweedfs.securityConfigEnabled" . }}
- name: security-config
mountPath: /etc/seaweedfs
readOnly: true
{{- end }}
{{- if .Values.global.seaweedfs.enableSecurity }}
{{- range list "ca" "client" "vector" }}
- name: {{ . }}-cert
mountPath: /usr/local/share/ca-certificates/{{ . }}
readOnly: true
{{- end }}
{{- end }}
volumes:
- name: tmp
emptyDir: {}
{{- if include "seaweedfs.securityConfigEnabled" . }}
- name: security-config
configMap:
name: {{ include "seaweedfs.fullname" . }}-security-config
{{- end }}
{{- if .Values.global.seaweedfs.enableSecurity }}
{{- range list "ca" "client" "vector" }}
- name: {{ . }}-cert
secret:
secretName: {{ include "seaweedfs.fullname" $ }}-{{ . }}-cert
{{- end }}
{{- end }}
{{- end }}
@@ -0,0 +1,22 @@
{{- if .Values.vector.enabled }}
apiVersion: v1
kind: Service
metadata:
name: {{ include "seaweedfs.componentName" (list . "vector") }}
namespace: {{ include "seaweedfs.namespace" . | quote }}
labels:
{{- include "seaweedfs.vector.labels" . | nindent 4 }}
spec:
clusterIP: None
selector:
{{- include "seaweedfs.vector.selectorLabels" . | nindent 4 }}
ports:
- name: grpc
port: {{ .Values.vector.port }}
targetPort: grpc
{{- if .Values.vector.metricsPort }}
- name: metrics
port: {{ .Values.vector.metricsPort }}
targetPort: metrics
{{- end }}
{{- end }}
@@ -0,0 +1,25 @@
{{- if and .Values.vector.enabled .Values.vector.metricsPort .Values.global.seaweedfs.monitoring.enabled }}
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
name: {{ include "seaweedfs.componentName" (list . "vector") }}
namespace: {{ include "seaweedfs.namespace" . | quote }}
labels:
{{- include "seaweedfs.vector.selectorLabels" . | nindent 4 }}
app.kubernetes.io/managed-by: {{ .Release.Service }}
helm.sh/chart: {{ .Chart.Name }}-{{ .Chart.Version | replace "+" "_" }}
{{- with include "seaweedfs.commonLabels" (dict "context" $ "exclude" $.Values.global.seaweedfs.monitoring.additionalLabels) }}{{ . | nindent 4 }}{{ end }}
{{- with .Values.global.seaweedfs.monitoring.additionalLabels }}{{ toYaml . | nindent 4 }}{{ end }}
{{- with .Values.vector.serviceMonitor.annotations }}
annotations:
{{- toYaml . | nindent 4 }}
{{- end }}
spec:
endpoints:
- port: metrics
interval: 30s
scrapeTimeout: 5s
selector:
matchLabels:
{{- include "seaweedfs.vector.selectorLabels" . | nindent 6 }}
{{- end }}
@@ -239,6 +239,22 @@ spec:
{{- /* the URL crosses a shell line; metacharacters must arrive as data, not syntax */}}
- name: LANCE_NAMESPACE_URL
value: {{ include "seaweedfs.worker.lanceNamespaceUrl" . | quote }}
{{- $mergedExtraEnvironmentVars := dict }}
{{- include "seaweedfs.mergeExtraEnvironmentVars" (dict "global" .Values.global.seaweedfs "component" .Values.worker "target" $mergedExtraEnvironmentVars) }}
{{- /* a duplicate would win over the chart's value and change --id or --namespace */}}
{{- $mergedExtraEnvironmentVars = omit $mergedExtraEnvironmentVars "POD_NAME" "LANCE_NAMESPACE_URL" }}
{{- range $key := keys $mergedExtraEnvironmentVars | sortAlpha }}
{{- $value := index $mergedExtraEnvironmentVars $key }}
- name: {{ $key }}
{{- if kindIs "string" $value }}
value: {{ tpl $value $ | quote }}
{{- else if kindIs "map" $value }}
valueFrom:
{{ toYaml $value | nindent 16 | trim }}
{{- else if not (kindIs "invalid" $value) }}
value: {{ toJson $value | quote }}
{{- end -}}
{{- end }}
command:
- "/bin/sh"
- "-ec"
@@ -256,6 +272,10 @@ spec:
--admin={{ template "seaweedfs.fullname" . }}-admin.{{ include "seaweedfs.namespace" . }}:{{ .Values.admin.port }}{{ if .Values.admin.grpcPort }}.{{ .Values.admin.grpcPort }}{{ end }} \
{{- end }}
--namespace="$LANCE_NAMESPACE_URL" \
--masters={{ include "seaweedfs.masterServerArg" . }} \
{{- if .Values.filer.filerGroup }}
--filer-group={{ .Values.filer.filerGroup }} \
{{- end }}
{{- if .Values.global.seaweedfs.enableSecurity }}
--tls-ca=/usr/local/share/ca-certificates/ca/tls.crt \
--tls-cert=/usr/local/share/ca-certificates/worker/tls.crt \
@@ -268,10 +288,20 @@ spec:
--max-concurrency={{ .Values.worker.maxExecute }}
volumeMounts:
{{- include "seaweedfs.tmpDirVolumeMount" (list . .Values.worker.containerSecurityContext "" "") | nindent 12 }}
{{- if include "seaweedfs.securityConfigEnabled" . }}
- name: security-config
readOnly: true
mountPath: /etc/seaweedfs/security.toml
subPath: security.toml
{{- end }}
{{- if .Values.global.seaweedfs.enableSecurity }}
- name: ca-cert
readOnly: true
mountPath: /usr/local/share/ca-certificates/ca/
{{- /* grpc.client, which the worker dials the masters and filers with */}}
- name: client-cert
readOnly: true
mountPath: /usr/local/share/ca-certificates/client/
- name: worker-cert
readOnly: true
mountPath: /usr/local/share/ca-certificates/worker/
@@ -0,0 +1,113 @@
package tests
import (
"fmt"
"os"
"os/exec"
"path/filepath"
"strings"
"testing"
)
func TestVectorBucketHook(t *testing.T) {
for _, auth := range []bool{false, true} {
t.Run(fmt.Sprint(auth), func(t *testing.T) {
output, documents := render(t, "s3.enabled=true", "vector.enabled=true", "vector.createBuckets[0]=embeddings", "s3.enableAuth="+fmt.Sprint(auth), "networkPolicy.enabled=true")
var found bool
for _, document := range documents {
metadata := document["metadata"].(map[string]interface{})
if document["kind"] == "Job" && strings.HasSuffix(metadata["name"].(string), "-vector-bucket-hook") {
found = true
if metadata["annotations"].(map[string]interface{})["helm.sh/hook"] != "post-install,post-upgrade" {
t.Fatal("missing upgrade hook")
}
}
}
if !found || !strings.Contains(output, "s3vectors get-vector-bucket") || !strings.Contains(output, "s3vectors create-vector-bucket") {
t.Fatal("missing vector bucket provisioning")
}
if auth && !strings.Contains(output, "key: \"admin_access_key_id\"") {
t.Fatal("missing generated admin credential reference")
}
})
}
}
func TestVectorBucketHookRetriesRacesAndRefusesConflicts(t *testing.T) {
_, documents := render(t, "s3.enabled=true", "vector.enabled=true", "vector.createBuckets[0]=embeddings")
var script string
for _, document := range documents {
if document["kind"] != "Job" {
continue
}
metadata := document["metadata"].(map[string]interface{})
if strings.HasSuffix(metadata["name"].(string), "-vector-bucket-hook") {
pod := document["spec"].(map[string]interface{})["template"].(map[string]interface{})["spec"].(map[string]interface{})
container := pod["containers"].([]interface{})[0].(map[string]interface{})
script = container["args"].([]interface{})[0].(string)
}
}
if script == "" {
t.Fatal("missing hook script")
}
for _, mode := range []string{"missing", "existing", "race", "iceberg", "denied"} {
t.Run(mode, func(t *testing.T) {
dir := t.TempDir()
stub := `#!/bin/sh
shift 2
operation=$2
case "$operation" in
get-vector-bucket)
if [ "$BUCKET_MODE" = denied ]; then echo AccessDenied >&2; exit 1; fi
if [ "$BUCKET_MODE" = existing ] || { [ -f "$BUCKET_STATE" ] && [ "$BUCKET_MODE" != iceberg ]; }; then exit 0; fi
echo NotFoundException >&2; exit 1;;
create-vector-bucket)
touch "$BUCKET_STATE"
if [ "$BUCKET_MODE" = race ] || [ "$BUCKET_MODE" = iceberg ]; then exit 1; fi;;
esac
`
if err := os.WriteFile(filepath.Join(dir, "aws"), []byte(stub), 0755); err != nil {
t.Fatal(err)
}
command := exec.Command("sh", "-ec", script)
command.Env = append(os.Environ(), "PATH="+dir+":"+os.Getenv("PATH"), "S3_ENDPOINT=http://unused", "BUCKET_MODE="+mode, "BUCKET_STATE="+filepath.Join(dir, "state"))
output, err := command.CombinedOutput()
wantFailure := mode == "iceberg" || mode == "denied"
if (err != nil) != wantFailure {
t.Fatalf("mode %s: %v %s", mode, err, output)
}
})
}
}
func TestVectorBucketHookRejectsShellNames(t *testing.T) {
if _, err := exec.LookPath("helm"); err != nil {
t.Skip("helm is not installed")
}
command := exec.Command("helm", "template", "test", "..", "--set", "s3.enabled=true", "--set", "vector.enabled=true", "--set", "vector.createBuckets[0]=bucket;touch /tmp/invalid")
output, err := command.CombinedOutput()
if err == nil || !strings.Contains(string(output), "vector.createBuckets names") {
t.Fatalf("invalid bucket accepted: %v %s", err, output)
}
}
func TestVectorBucketHookGatewayModes(t *testing.T) {
for _, mode := range []string{"s3", "filer", "allInOne"} {
settings := []string{"vector.enabled=true", "vector.createBuckets[0]=embeddings"}
component := "s3"
switch mode {
case "s3":
settings = append(settings, "s3.enabled=true", "s3.enableAuth=true")
case "filer":
settings = append(settings, "s3.enabled=false", "filer.s3.enabled=true", "filer.s3.enableAuth=true")
component = "filer"
case "allInOne":
settings = append(settings, "s3.enabled=false", "allInOne.enabled=true", "allInOne.s3.enabled=true", "allInOne.s3.enableAuth=true")
component = "all-in-one"
}
output, _ := render(t, settings...)
if !strings.Contains(output, "http://vector-test-seaweedfs-"+component+".default:8333") || !strings.Contains(output, "key: \"admin_access_key_id\"") {
t.Fatalf("missing endpoint or auth for %s", mode)
}
}
}
+148
View File
@@ -0,0 +1,148 @@
package tests
import (
"fmt"
"io"
"os"
"os/exec"
"strings"
"testing"
"go.yaml.in/yaml/v3"
)
func render(t *testing.T, settings ...string) (string, []map[string]interface{}) {
t.Helper()
if _, err := exec.LookPath("helm"); err != nil {
t.Skip("helm is not installed")
}
args := []string{"template", "vector-test", ".."}
for _, setting := range settings {
args = append(args, "--set", setting)
}
output, err := exec.Command("helm", args...).CombinedOutput()
if err != nil {
t.Fatalf("helm: %v\n%s", err, output)
}
decoder := yaml.NewDecoder(strings.NewReader(string(output)))
var documents []map[string]interface{}
for {
var document map[string]interface{}
if err := decoder.Decode(&document); err == io.EOF {
break
} else if err != nil {
t.Fatal(err)
}
if document != nil {
documents = append(documents, document)
}
}
return string(output), documents
}
func TestVectorDeployment(t *testing.T) {
_, defaults := render(t)
for _, document := range defaults {
if strings.HasSuffix(fmt.Sprint(document["metadata"].(map[string]interface{})["name"]), "-vector") {
t.Fatal("vector resource enabled by default")
}
}
for _, secure := range []bool{false, true} {
t.Run(fmt.Sprintf("security-%v", secure), func(t *testing.T) {
output, documents := render(t, "vector.enabled=true", "vector.metricsPort=9329", "vector.imageOverride=example/metrics-engine:test", "vector.port=8098", "global.seaweedfs.enableSecurity="+fmt.Sprint(secure), "global.seaweedfs.monitoring.enabled=true", "networkPolicy.enabled=true")
for _, want := range []string{"--listen=\"$POD_IP:8098\"", "--metrics-port=9329", "path: /ready", "path: /health", "kind: ServiceMonitor"} {
if !strings.Contains(output, want) {
t.Fatalf("missing %s", want)
}
}
found := false
for _, document := range documents {
metadata := document["metadata"].(map[string]interface{})
if document["kind"] == "Deployment" && strings.HasSuffix(metadata["name"].(string), "-vector") {
found = true
spec := document["spec"].(map[string]interface{})["template"].(map[string]interface{})["spec"].(map[string]interface{})
if spec["nodeSelector"].(map[string]interface{})["kubernetes.io/os"] != "linux" {
t.Fatal("vector engine must use Linux image platforms")
}
}
}
if !found {
t.Fatal("no vector deployment")
}
if secure && (!strings.Contains(output, "[grpc.vector]") || !strings.Contains(output, "vector-cert")) {
t.Fatal("missing vector TLS configuration")
}
})
}
}
func TestVectorExistingImagesAndExternalCertificates(t *testing.T) {
output, _ := render(t, "vector.enabled=true")
start := strings.Index(output, "# Source: seaweedfs/templates/vector/vector-deployment.yaml")
deployment := strings.Split(output[start:], "\n---")[0]
if strings.Contains(deployment, "--metrics-port") || !strings.Contains(deployment, "tcpSocket:") {
t.Fatal("default probes must support existing vector images")
}
output, _ = render(t, "vector.enabled=true", "global.seaweedfs.enableSecurity=true", "certificates.externalCertificates.enabled=true")
if strings.Contains(output, "kind: Certificate") || !strings.Contains(output, "mountPath: /usr/local/share/ca-certificates/vector") {
t.Fatal("external vector certificates are not mounted")
}
}
func TestVectorIPv6Listen(t *testing.T) {
_, docs := render(t, "vector.enabled=true")
var args []interface{}
for _, doc := range docs {
if doc["kind"] != "Deployment" {
continue
}
pod := doc["spec"].(map[string]interface{})["template"].(map[string]interface{})["spec"].(map[string]interface{})
c := pod["containers"].([]interface{})[0].(map[string]interface{})
if c["name"] == "vector" {
args = c["args"].([]interface{})
}
}
for _, ip := range []string{"10.0.0.20", "fd00::20"} {
dir := t.TempDir()
script := strings.ReplaceAll(args[0].(string), "/usr/bin/weed-vector", "printf '%s\\n'")
cmd := exec.Command("sh", "-ec", script, "vector", "--masters=master")
cmd.Dir = dir
cmd.Env = append(os.Environ(), "POD_IP="+ip)
out, err := cmd.CombinedOutput()
address := ip
if strings.Contains(ip, ":") {
address = "[" + ip + "]"
}
if err != nil || !strings.Contains(string(out), "--listen="+address+":8090") || !strings.Contains(string(out), "--masters=master") {
t.Fatalf("%s: %s %v", ip, out, err)
}
}
}
func TestVectorAllInOneMasters(t *testing.T) {
output, _ := render(t, "vector.enabled=true", "allInOne.enabled=true", "master.enabled=false")
if !strings.Contains(output, "--masters=vector-test-seaweedfs-all-in-one.default:9333") {
t.Fatal("vector cannot discover all-in-one master")
}
output, _ = render(t, "vector.enabled=true", "allInOne.enabled=true", "master.enabled=false", "vector.masters=external:9333")
if !strings.Contains(output, "--masters=external:9333") {
t.Fatal("explicit master override lost")
}
}
func TestVectorNetworkPolicyOverrides(t *testing.T) {
output, _ := render(t, "vector.enabled=true", "s3.enabled=true", "vector.createBuckets[0]=embeddings", "networkPolicy.enabled=true", "networkPolicy.components.vector.extraIngress[0].ports[0].port=9329", "networkPolicy.egress.enabled=true", "networkPolicy.components.vector-bucket-hook.extraEgress[0].ports[0].port=443")
if !strings.Contains(output, "port: 9329") || !strings.Contains(output, "port: 443") {
t.Fatal("vector network overrides missing")
}
}
func TestVectorMetricsImageCompatibility(t *testing.T) {
if _, err := exec.LookPath("helm"); err != nil {
t.Skip("helm is not installed")
}
out, err := exec.Command("helm", "template", "vector-test", "..", "--set", "vector.enabled=true", "--set", "vector.metricsPort=9329").CombinedOutput()
if err == nil || !strings.Contains(string(out), "default 4.53 image does not support metrics") {
t.Fatalf("incompatible image accepted: %v %s", err, out)
}
render(t, "vector.enabled=true", "vector.metricsPort=9329", "vector.imageOverride=example/metrics-engine:test")
}
+74
View File
@@ -1083,12 +1083,16 @@ filer:
# You may specify buckets to be created during the install or upgrade process.
# Buckets may be exposed publicly by setting `anonymousRead` to `true`
# ttl format: [1-255][m|h|d|w|M|y] (e.g., 7d)
# replication (e.g., 001) and disk (e.g., ssd) apply to writes into the bucket
# removing ttl, replication or disk later keeps the stored rule; set a new value instead
# objectLock enables S3 Object Lock (irreversible, forces versioning)
# versioning: Enabled or Suspended (or bool true/false)
# createBuckets:
# - name: bucket-a
# anonymousRead: true
# ttl: 7d
# replication: "001"
# disk: ssd
# objectLock: true
# versioning: Enabled
# - name: bucket-b
@@ -1165,12 +1169,16 @@ s3:
# You may specify buckets to be created during the install or upgrade process.
# Buckets may be exposed publicly by setting `anonymousRead` to `true`
# ttl format: [1-255][m|h|d|w|M|y] (e.g., 7d)
# replication (e.g., 001) and disk (e.g., ssd) apply to writes into the bucket
# removing ttl, replication or disk later keeps the stored rule; set a new value instead
# objectLock enables S3 Object Lock (irreversible, forces versioning)
# versioning: Enabled or Suspended (or bool true/false)
# createBuckets:
# - name: bucket-a
# anonymousRead: true
# ttl: 7d
# replication: "001"
# disk: ssd
# objectLock: true
# versioning: Enabled
# - name: bucket-b
@@ -1733,6 +1741,60 @@ admin:
serviceMonitor:
annotations: {}
vector:
enabled: false
imageOverride: null
replicas: 1
port: 8090
metricsPort: 0
masters: ""
readCacheMB: 512
nprobes: null
extraArgs: []
extraEnvironmentVars: {}
podLabels: {}
podAnnotations: {}
annotations: {}
nodeSelector:
kubernetes.io/os: linux
tolerations: []
affinity:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: kubernetes.io/arch
operator: In
values: [amd64, arm64]
resources: {}
podSecurityContext: {}
containerSecurityContext:
allowPrivilegeEscalation: false
capabilities:
drop: [ALL]
serviceMonitor:
annotations: {}
createBuckets: []
bucketHook:
podSecurityContext:
seccompProfile:
type: RuntimeDefault
containerSecurityContext:
allowPrivilegeEscalation: false
readOnlyRootFilesystem: true
capabilities:
drop: [ALL]
image: amazon/aws-cli:2.39.6
endpoint: ""
region: us-east-1
credentialsSecret: ""
accessKeyKey: AWS_ACCESS_KEY_ID
secretKeyKey: AWS_SECRET_ACCESS_KEY
sessionTokenKey: AWS_SESSION_TOKEN
caBundleSecret: ""
caBundleKey: tls.crt
resources: {}
worker:
enabled: false
imageOverride: null
@@ -1744,6 +1806,14 @@ worker:
# The lance_* jobs run in their own container, /usr/bin/weed-worker.
# amd64/arm64 only; pin mixed clusters with worker.affinity/nodeSelector.
# It keeps tables through the filers the masters list and reads
# security.toml. The Lance namespace knows it by a token derived from
# jwt.filer_signing.key, which the chart renders only with
# global.seaweedfs.securityConfig.jwtSigning.filerWrite: true. When S3 has
# identities, set it, or the namespace lists the worker no tables and the
# lance_* jobs find nothing to do. It also makes the filer accept only signed
# HTTP writes, so clients writing to the filer's HTTP API directly need a JWT
# signed with that key.
# Lance namespace URL override; empty derives it from s3.lancePort.
namespaceUrl: ""
@@ -1945,12 +2015,16 @@ allInOne:
# You may specify buckets to be created during the install process.
# Buckets may be exposed publicly by setting `anonymousRead` to `true`
# ttl format: [1-255][m|h|d|w|M|y] (e.g., 7d)
# replication (e.g., 001) and disk (e.g., ssd) apply to writes into the bucket
# removing ttl, replication or disk later keeps the stored rule; set a new value instead
# objectLock enables S3 Object Lock (irreversible, forces versioning)
# versioning: Enabled or Suspended (or bool true/false)
# createBuckets:
# - name: bucket-a
# anonymousRead: true
# ttl: 7d
# replication: "001"
# disk: ssd
# objectLock: true
# versioning: Enabled
# - name: bucket-b
+83
View File
@@ -0,0 +1,83 @@
<svg xmlns="http://www.w3.org/2000/svg" width="1080" height="650" viewBox="0 0 1080 650" role="img" aria-labelledby="title description">
<title id="title">SeaweedFS: one cluster for objects, files, tables, and vectors</title>
<desc id="description">S3 object storage, a POSIX file system, table buckets with a built-in Iceberg REST Catalog and Lance namespace, and S3 Vector Buckets share one SeaweedFS storage cluster. Masters coordinate the cluster, filers manage namespaces and metadata, and volume servers store replicated or erasure-coded blobs. Cloud caching, cloud tiering, and cross-cluster replication extend the cluster.</desc>
<defs>
<marker id="arrow" viewBox="0 0 10 10" refX="8" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse">
<path d="M 0 0 L 10 5 L 0 10 z" fill="#75928e"/>
</marker>
</defs>
<rect width="1080" height="650" rx="20" fill="#f5f9f8"/>
<g font-family="Arial, Helvetica, sans-serif">
<text x="40" y="43" fill="#0f766e" font-size="15" font-weight="700" letter-spacing="2">SEAWEEDFS</text>
<text x="40" y="86" fill="#163832" font-size="32" font-weight="700">One cluster. Objects, files, tables, vectors.</text>
<!-- Four ways to use the same storage cluster. -->
<g>
<rect x="40" y="116" width="238" height="184" rx="14" fill="#fff" stroke="#cbded8"/>
<g transform="translate(62 136)" fill="none" stroke="#187b67" stroke-width="2.2" stroke-linejoin="round">
<path d="m0 10 18-9 18 9-18 9z M0 10v21l18 9 18-9V10 M18 19v21"/>
</g>
<text x="62" y="207" fill="#163832" font-size="22" font-weight="700">Object storage</text>
<text x="62" y="237" fill="#48635e" font-size="18">S3 API</text>
<text x="62" y="261" fill="#48635e" font-size="17">Apps, backups, media</text>
</g>
<g>
<rect x="294" y="116" width="238" height="184" rx="14" fill="#fff" stroke="#cbded8"/>
<g transform="translate(316 140)" fill="none" stroke="#2676a3" stroke-width="2.2" stroke-linejoin="round">
<path d="M0 7V2h15l6 6h17v27H0V7z M0 12h38"/>
</g>
<text x="316" y="207" fill="#163832" font-size="22" font-weight="700">File system</text>
<text x="316" y="237" fill="#48635e" font-size="18">POSIX / FUSE</text>
<text x="316" y="261" fill="#48635e" font-size="17">WebDAV, SFTP, HDFS</text>
</g>
<g>
<rect x="548" y="116" width="238" height="184" rx="14" fill="#fff" stroke="#cbded8"/>
<g transform="translate(570 138)" fill="none" stroke="#7a62ab" stroke-width="2.2" stroke-linejoin="round">
<rect x="0" y="0" width="38" height="36" rx="3"/>
<path d="M0 12h38 M0 24h38 M13 12v24 M26 12v24"/>
</g>
<text x="570" y="207" fill="#163832" font-size="22" font-weight="700">Table buckets</text>
<text x="570" y="237" fill="#48635e" font-size="18">S3 Tables</text>
<text x="570" y="261" fill="#48635e" font-size="17">Iceberg REST Catalog</text>
<text x="570" y="285" fill="#48635e" font-size="17">Lance namespace</text>
</g>
<g>
<rect x="802" y="116" width="238" height="184" rx="14" fill="#e4f5ee" stroke="#187b67" stroke-width="1.5"/>
<g transform="translate(824 138)" fill="none" stroke="#187b67" stroke-width="2.2">
<path d="m5 29 13-22 18 22H5z M18 7v30 M5 29l13 8 18-8"/>
<g fill="#e4f5ee">
<circle cx="5" cy="29" r="4"/>
<circle cx="18" cy="7" r="4"/>
<circle cx="36" cy="29" r="4"/>
<circle cx="18" cy="37" r="4"/>
</g>
</g>
<text x="824" y="207" fill="#163832" font-size="22" font-weight="700">Vector buckets</text>
<text x="824" y="237" fill="#266354" font-size="18">S3 Vectors</text>
<text x="824" y="261" fill="#266354" font-size="17">Search, RAG, embeddings</text>
</g>
<!-- A shared connection expresses capabilities, not request routing. -->
<path d="M159 300v28h762v-28 M413 300v28 M667 300v28 M540 328v30" fill="none" stroke="#75928e" stroke-width="2" marker-end="url(#arrow)"/>
<rect x="40" y="362" width="1000" height="174" rx="16" fill="#163e35"/>
<text x="540" y="400" text-anchor="middle" fill="#fff" font-size="25" font-weight="700">One shared storage cluster</text>
<g fill="#fff" font-size="20" font-weight="700" text-anchor="middle">
<text x="213" y="445">Masters</text>
<text x="540" y="445">Filers</text>
<text x="867" y="445">Volume servers</text>
</g>
<path d="M376 425v51 M704 425v51" stroke="#497266"/>
<g fill="#cfe4dc" font-size="17" text-anchor="middle">
<text x="213" y="473">Coordination &amp; placement</text>
<text x="540" y="473">Namespaces &amp; metadata</text>
<text x="867" y="473">Replicated or erasure-coded blobs</text>
</g>
<text x="540" y="511" text-anchor="middle" fill="#cfe4dc" font-size="17">Add volume servers to grow capacity</text>
<path d="M284 536v23 M796 536v23" stroke="#75928e" stroke-width="2" stroke-dasharray="5 5"/>
<rect x="40" y="560" width="488" height="58" rx="12" fill="#fff" stroke="#cbded8"/>
<rect x="552" y="560" width="488" height="58" rx="12" fill="#fff" stroke="#cbded8"/>
<text x="284" y="596" text-anchor="middle" fill="#305c50" font-size="19">Cloud caching &amp; tiering</text>
<text x="796" y="596" text-anchor="middle" fill="#305c50" font-size="19">Cross-cluster replication</text>
</g>
</svg>

After

Width:  |  Height:  |  Size: 5.3 KiB

+1084 -1008
View File
File diff suppressed because it is too large. Load diff

Before

Width:  |  Height:  |  Size: 53 KiB

After

Width:  |  Height:  |  Size: 55 KiB

@@ -286,6 +286,7 @@ message WriteCondition {
IF_EXTENDED_TIME_ELAPSED = 8; // fail if present and extended[ext_key] (unix seconds) is in the future
IF_CHUNKS_EQUAL = 9; // fail unless the stored chunk fid multiset equals fids (absent entry = no chunks)
IF_ENTRY_EQUAL = 10; // fail unless the stored entry equals expected_entry
IF_OBJECT_EXISTS = 11; // fail if absent or a plain directory, one holding no object of its own (If-Match: * on a null version)
}
// Clause is one primitive comparison. IF_ETAG_MATCH holds when the current
// entry's ETag equals any value in etags; IF_ETAG_NOT_MATCH holds when it
File diff suppressed because it is too large. Load diff
+3
View File
@@ -6,6 +6,9 @@
//! second copy is a correctness risk rather than a typing cost: the HTTP↔gRPC
//! address rule, which two copies had already drifted on, and the process-wide
//! rustls provider, which only works if every binary installs the same one.
//! The security.toml search path joined them for the same reason: an operator
//! mounts the file once and every binary has to look where `weed` looks.
pub mod address;
pub mod security_config;
pub mod tls;
+79
View File
@@ -0,0 +1,79 @@
//! Where a SeaweedFS process finds security.toml, the way `weed` does.
//!
//! Mirrors the search list in `weed/util/config.go`: the working directory,
//! `$HOME/.seaweedfs`, `/usr/local/etc/seaweedfs` and `/etc/seaweedfs`, first
//! existing file wins. The volume server and the workers each read the file
//! for different sections, but an operator mounts it in one place and expects
//! every binary to pick it up from there.
use std::ffi::OsString;
use std::path::{Path, PathBuf};
pub const SECURITY_CONFIG_FILE_NAME: &str = "security.toml";
/// The locations searched for security.toml, in order.
pub fn candidates(current_dir: Option<&Path>, home_dir: Option<&Path>) -> Vec<PathBuf> {
let mut candidates = Vec::new();
if let Some(dir) = current_dir {
candidates.push(dir.join(SECURITY_CONFIG_FILE_NAME));
}
if let Some(home) = home_dir {
candidates.push(home.join(".seaweedfs").join(SECURITY_CONFIG_FILE_NAME));
}
candidates.push(PathBuf::from("/usr/local/etc/seaweedfs").join(SECURITY_CONFIG_FILE_NAME));
candidates.push(PathBuf::from("/etc/seaweedfs").join(SECURITY_CONFIG_FILE_NAME));
candidates
}
/// The first candidate that is a file.
pub fn find(current_dir: Option<&Path>, home_dir: Option<&Path>) -> Option<PathBuf> {
candidates(current_dir, home_dir)
.into_iter()
.find(|candidate| candidate.is_file())
}
/// `$HOME`, or `%USERPROFILE%` where that is what the platform sets, read
/// through an injected lookup so tests need not touch the process environment.
pub fn home_dir(env: &dyn Fn(&str) -> Option<OsString>) -> Option<PathBuf> {
env("HOME")
.filter(|v| !v.is_empty())
.map(PathBuf::from)
.or_else(|| {
env("USERPROFILE")
.filter(|v| !v.is_empty())
.map(PathBuf::from)
})
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn search_order_matches_go() {
let found = candidates(Some(Path::new("/work")), Some(Path::new("/home/u")));
assert_eq!(
found,
vec![
PathBuf::from("/work/security.toml"),
PathBuf::from("/home/u/.seaweedfs/security.toml"),
PathBuf::from("/usr/local/etc/seaweedfs/security.toml"),
PathBuf::from("/etc/seaweedfs/security.toml"),
]
);
}
#[test]
fn home_falls_back_to_userprofile() {
let env = |key: &str| match key {
"USERPROFILE" => Some(OsString::from("/users/u")),
_ => None,
};
assert_eq!(home_dir(&env), Some(PathBuf::from("/users/u")));
let empty_home = |key: &str| match key {
"HOME" => Some(OsString::new()),
_ => None,
};
assert_eq!(home_dir(&empty_home), None);
}
}
+798 -10
View File
File diff suppressed because it is too large. Load diff
+339
View File
@@ -0,0 +1,339 @@
# S3 Vectors ANN — Dev Plan
## Goal
Approximate nearest-neighbour search for S3 Vectors indexes, built on Lance
vector indexes, so that query latency stops growing linearly with the index.
Target: 90%+ recall@10 (what AWS states for S3 Vectors), exact results below
a size threshold and for rows written since the last index build, and one
code path that serves both the open-source single engine and the Enterprise
distributed query layer. Writes stay unchanged.
## Status (2026-11-04)
| Phase | Branch | State |
|---|---|---|
| 0 Build and measure | `vector-ann-phase0` | open: `vector_index` builder, `weed-vector-cli index`, `PERF_INDEX`, `PERF_CLUSTERS` |
| 1 Probe policy | `vector-ann-phase1` | open: policy in `query::scanner`, `--nprobes`, one Lance session per engine |
| 2 Build job | `vector-ann-phase2` | open: `lance_vector_index` in `weed-worker`, admin-driven end-to-end test, worker CI legs |
| 3 Keep it covering | `vector-ann-phase3` | open: relative budget, retrain rule, end-to-end steps |
| 4 Docs and gates | — | after the above merge |
| 5 Enterprise sharded builds | — | the Enterprise repository |
What the measurements changed:
- **IVF needs structure in the data.** On independent Gaussian vectors, the
perf suite's default, recall through an IVF index tracks the fraction of
partitions probed: at 50k × 384 with 224 partitions, 8 probes gave 0.13 and
64 probes 0.56, and latency did not fall with fewer probes. On vectors drawn
around 100 shared centres, 8 probes gave recall 1.0; with eight clients
the index served 145 queries/s against 52 exact at 50k and 70 against 11
at 200k, and the 10% filtered query at 200k fell from 445 to 265 ms,
while one client's latency stayed at the request path's floor on the
shared host. Embeddings cluster; the perf suite grows `PERF_CLUSTERS`
and the end-to-end test draws its rows around centres. The probe rule
stands as written.
- **The engine opened every table in a fresh Lance session**, so the index
cache died with each request. Phase 1 gives the engine one session, sized
by `--index-cache-mb`.
- **Compaction keeps the index covering** (Lance 10 remaps it), so the
compaction risk below did not materialise.
## Where things stand (2026-11-04)
What already exists, with the files that hold it:
- **The engine is index-aware; nothing builds an index.** `QueryVectors` in
`crates/server/src/service.rs` always calls `weed_vector_core::query::ann`,
which runs a Lance scan with `use_index(true)`: ANN on fragments an index
covers, a flat pass on the rest, decided per fragment by Lance
(`crates/core/src/query.rs`). `nprobes` is `None`. `CreateIndex` writes an
empty Lance dataset and no vector index, so today every fragment is "tail"
and every query is an exact scan.
- **The maintenance worker only extends indexes that exist.** The
`lance_optimize_indices` job in `seaweed-worker/crates/lance/src/jobs/indices.rs`
skips a table whose `load_indices()` is empty and otherwise calls
`optimize_indices(&OptimizeOptions::default())` once `num_unindexed_rows`
exceeds its `max_unindexed_rows` budget (default 100,000).
- **Coverage and segments are modelled.** `crates/core/src/coverage.rs` reports,
per index name, the fragments covered and the uncovered rows; it requires the
fragment bitmap each Lance `IndexMetadata` delta carries.
`crates/core/src/segments.rs` lists physical segments (one per delta) with
their names and bitmaps, and partitions them across workers. `weed-vector-cli
coverage` and `weed-vector-cli query --flat` expose both.
- **Filters are applied before ranking.** `query.rs` sets `prefilter(true)`
unless the request asks for post-filtering, which S3 Vectors never does.
Lance's IVF search honours the prefilter, so this is already the behaviour
AWS calls an `ENHANCED` index.
- **Enterprise consumes exactly this model.** In the Enterprise repository,
`enterprise/weed/pb/vector_coord.proto` defines `SegmentInfo{id,
resident_bytes, rows, fragment_ids}` and `SegmentList{tail_fragment_ids,
tail_bytes}`; `VectorQuery` already carries `nprobes`; a `Task` is a set of
segment ids plus fragment ids. `sw-vector-distributed/src/plan.rs` assigns
a fragment covered by several segments to the lowest segment and never mixes
index names in one scan; `resident_bytes` is the index files' size.
`datasets.rs` warms a slot with `prewarm_index_segments` per index name and
warms the tail by scanning it. `sw-vector-distributed/src/index.rs` holds
`plan_index` (shard the uncovered fragments across workers),
`build_index_task` (one delta per shard, named
`<column>-w<worker>-v<version>`, built with `create_index_builder(..)
.fragments(..)`) and `optimize_policy` (append deltas until `MAX_DELTAS = 8`,
then Lance's default options, which promise no merge; see phase 3).
Nothing calls these three yet; `tests/index.rs`,
`tests/plan_search.rs` and `tests/engine_query.rs` exercise them with
`ivf_flat(4)` indexes.
- **Lance 10.0.0 offers** `VectorIndexParams::ivf_flat`, `ivf_pq`,
`with_ivf_sq_params`, `with_ivf_rq_params`, `ivf_hnsw` and HNSW variants;
scanner knobs `nprobes`, `minimum_nprobes`, `maximum_nprobes`, `ef`,
`refine`; `OptimizeOptions{num_indices_to_merge, index_names, retrain}`.
- **Measured today** (384-d, cosine, top-10, one 16-core host, exact scans):
one engine answers in 42 ms at 50k vectors and 82 ms at 200k; filtered
queries take 81 and 259–336 ms. Four Enterprise engines split four ways
bring 200k to 58 ms and the filtered query to 73 ms. Recall is 1.000.
- **Tests assume exactness only on tiny tables.** `test/s3vectors/s3vectors_test.py`
asserts exact result order on indexes of a handful of vectors; the perf
suite asserts recall@10 ≥ 0.9 against a client-side exact search.
- **AWS**: "90%+ average recall for most datasets"; `CLASSIC` and `ENHANCED`
index modes, the latter pre-filtering; no algorithm named.
## Design
### The contract every consumer relies on
An S3 Vectors index gains a Lance vector index on its `vector` column, and
everything downstream keeps working because it only ever sees:
1. **Segments**: Lance index deltas with fragment bitmaps. Every build and
every merge must leave `coverage_known == true`; a delta without a bitmap
turns the whole index into "rebuild" for the Enterprise planner.
2. **Names**: one logical index name per shard, never two shards under one
name within a build. Scans and warm-ups group by name.
3. **Tail**: fragments no segment covers, searched exactly. Rows written
after a build land here, so recall never drops to zero between builds.
4. **Bytes**: `resident_bytes` of a segment is its index files' size. The
controller's W sizing (`W_mem`, `W_scan`) and the engine's
`--capacity-bytes` budget use it unchanged.
5. **Versions**: a build or merge commits a new dataset version. Routers cache
segment lists per version, plans pin versions with the lease tags in
`crates/core/src/maintenance.rs`, and cleanup respects those leases. No
new mechanism is needed, but the build job must go through the same
commit path as any Lance write.
### Index type and parameters
- **IVF_FLAT first.** Distances inside probed partitions are exact, so recall
is governed by `nprobes` alone; the only training is k-means over the
centroids; resident bytes stay close to the raw vectors, which keeps the
Enterprise W sizing in the same range as today. `IVF_PQ` becomes an
operator option later for indexes whose raw size is the constraint. HNSW
variants are out of scope: their build cost and memory do not fit the
maintenance-worker model.
- **Metric from the table.** The vector field's `distance_metric` metadata
(`COSINE`, `DOT_PRODUCT`, else L2) is the metric the index is built with,
exactly as `service.rs` resolves it for queries.
- **Partitions**: start at `clamp(round(sqrt(rows)), 1, 4096)` for the whole
table, and for a shard, `sqrt` of the shard's rows. Phase 0 measures
recall and latency against this and adjusts before it becomes a default.
- **Probes**: one count per query, `max(8, ceil(partitions / 10))` clamped to
the partition count, taken from the largest delta of the index. A delta
with no more partitions than that count is searched in full; a larger one
at that count. Lance searches at least that many partitions. Under a
filter it may widen to four times the count until it has k rows, which a
selective filter needs; an open upper bound made Lance prepare every
partition for that and cost as much as the exact scan (201 ms against
104 ms for the 10% filter at 50k × 384). Without a filter it stops at the
count.
### Who builds, and when
The Lance maintenance worker (`weed-worker`) gets a new job type,
`lance_vector_index`, beside `lance_optimize_indices`:
- **Detection**: tables in a vector bucket's `vector` namespace (ids of the
form `<bucket>$vector$<index>`) whose vector column has no index and at
least `min_rows` live rows (default 50,000). Below the threshold the index
stays exact, which also keeps the existing small-table tests exact.
- **Execution**: one `create_index` call with IVF_FLAT, the table's metric,
the partition rule above, and a fixed name (`vector_idx`). Progress and
counters follow the pattern of `indices.rs`; a new `indexes_built_total`
metric and an admin observation of index coverage sit beside the row and
fragment counts the worker already reports.
- **Why the worker and not the engine or the gateway.** `CreateIndex` has no
rows to index. The engine serves queries and should not run minute-long
k-means builds. The worker already has the catalog client, credentials,
the admin UI settings form, metrics, retries and dedupe, and plugin jobs
start disabled until an operator turns them on, which is the rollout
switch we want.
- **Enterprise sharded builds** (phase 5): the same job, when Enterprise
engines are registered, splits the uncovered fragments with `plan_index`
and has each engine run `build_index_task` for its shard, producing one
named delta per shard. This is what `index.rs` was written for; the worker
drives it, the engines build, and the shared `optimize_policy` keeps each
name's delta count under eight.
### Keeping it covering
`lance_optimize_indices` already does the work once an index exists. Two
changes:
- The budget becomes relative for vector tables: reindex when uncovered rows
exceed `min(absolute budget, 10% of indexed rows)`, so a 60k-row index
reindexes after 6k new rows instead of waiting for 100k.
- A retrain once the table has grown to about 4× the rows the centroids were
trained on, so partitions do not go stale. No row count is recorded: the
retrain is due when the partition rule for the current rows gives twice
the partitions the index has, and it rebuilds a single IVF_FLAT name
through the builder at that count. Past 2,048 partitions (about 4.2M
rows) the rule cannot double under the 4,096 cap, so those centroids are
not retrained again: that would need the trained row count recorded and a
full rebuild of a table that size, which is left for later.
### Deletes and upserts
A delete marks rows in Lance deletion files; an upsert is a delete plus an
append. Deleted rows are excluded through the prefilter on indexed and
unindexed fragments alike, and appended rows are tail until the next
optimize. Phase 1 adds a test that proves both, since this is where ANN
systems silently return stale hits.
### API surface
No new S3 Vectors API parameter is needed to reach AWS parity. Two optional
items for later: accept and echo AWS's `indexMode`/`queryMode` (both behave
as `ENHANCED` here, because filters are always applied first), and a
non-AWS per-request `nprobes` for the perf suite and the CLI, carried in the
engine's `QueryVectorsRequest` (the Enterprise `VectorQuery` already has the
field).
### Enterprise compatibility requirements
These hold for every phase, and the mono s3vectors legs (go/rust ×
single/distributed) are the gate:
- Builds produce deltas with fragment bitmaps; merges keep them. A test in
`tests/index.rs` asserts `coverage_known` after build, append and merge.
- Index names never mix within a shard set; the router and `plan.rs` group by
name, and `with_index_segments` rejects mixed names.
- Segment count per index name stays under `MAX_DELTAS`: at the limit
`optimize_policy(deltas)` passes `OptimizeOptions::merge(deltas)`, where
`deltas` is the name's delta count, so every delta of that name folds
into one with the new rows; Lance's default options may append another
delta instead of merging. A test runs repeated append-and-optimize cycles
and checks the actual segment counts.
- Coverage is measured as the union over an index's names. After a sharded
build each name misses the other shards' fragments, so a per-name count
would extend every shard over the whole table and duplicate the index.
An optimize extends one name only, the one with the fewest deltas; a test
checks that two disjoint shards stay disjoint through it.
- `resident_bytes` is the index files' size and `tail_bytes` the uncovered
fragments' size; the controller's `max_scan_mb` and memory sizing need no
change for IVF_FLAT, and PQ only makes slots smaller.
- A merge replaces segment UUIDs, which is a new placement epoch. The
existing make-before-break (target groups warm with
`prewarm_index_segments` while serving groups keep answering) covers it;
phase 5 measures how long a merged segment takes to warm and sets
expectations for `warmup_timeout`.
- Workers read index files the way they read data files, directly from the
filers and volume servers, and the files are write-once,
so the read cache and the warm path apply unchanged.
- Query semantics are identical in both modes: each engine applies the same
probe policy to the segments it owns, or the count the router passes in
`VectorQuery.nprobes`, plus a flat pass over its tail fragments; the
gateway merges the partial top-k lists. Each delta is probed on its own
centroids, so the merged answer is never worse than one engine's.
- The open-source `weed-vector` stays the writer and the fallback engine, and
it serves ANN through the same `query::ann` the workers call.
## Phases
### Phase 0 — Measure before deciding (small)
- `weed-vector-cli index build --partitions N --name ..` on a local table,
reusing the code paths the tests already use, so builds can be tried
without the worker.
- Perf suite: `PERF_INDEX=1` builds the index after ingest and reports exact
versus ANN latency and recall@10 at 50k, 200k and 1M × 384, unfiltered and
with the 10% filter, for a grid of partitions and `nprobes`.
- Verify Lance behaviour we depend on: deletions excluded under ANN;
compaction (`lance_compact`) keeps bitmaps and coverage; `optimize_indices`
with `append` versus merge versus `retrain`.
- Exit: chosen defaults for `min_rows`, partitions and probes with numbers
behind them, and a written answer to each verification item.
### Phase 1 — Engine query path (small)
- Probe policy when an index exists: `minimum_nprobes` is the per-query
count from the Probes rule above, taken from the largest delta;
`maximum_nprobes` is four times it, which Lance uses only under a
prefilter.
- `nprobes` in `QueryVectorsRequest` for tests and the CLI; zero means the
policy.
- Per-query log line and counters: segments searched, tail rows scanned,
index hit or not. Without these, recall regressions are invisible.
- Tests in `crates/core`: ANN versus flat recall on a 50k synthetic table;
delete and upsert correctness under ANN; filtered query with a selective
filter still returns k results.
### Phase 2 — Build job in the maintenance worker (medium)
- `lance_vector_index` job: detection, execution, settings form (`min_rows`,
index type, partition rule), metrics, admin observation of coverage.
- Unit tests after the pattern of the other jobs; an integration test in
`test/s3vectors` that ingests above the threshold, runs the worker, and
asserts coverage through the CLI and recall ≥ 0.9 through the API.
- Exit: a fresh cluster with the job enabled indexes a 100k-vector table on
its own, and the small-table tests stay exact.
### Phase 3 — Keep it covering (small)
- Relative budget and `retrain` policy in `lance_optimize_indices` for vector
tables. The retrain rebuilds through the builder with partitions for the
current row count; Lance's `OptimizeOptions::retrain()` does not size them.
- Union-based coverage over an index's names, extension scoped to one name,
and `optimize_policy` with an explicit merge of all deltas at
`MAX_DELTAS`, shared with the Enterprise builds in the worker crate.
- Tests: append and optimize nine times and check the delta counts climb to
the limit and fold; two names on disjoint fragments stay disjoint through
an optimize; the end-to-end test runs the job after a tenth of the index
is appended and again after the table has grown fivefold.
### Phase 4 — Gateway, docs and gates (small)
- Optional `indexMode`/`queryMode` acceptance; `GetIndex` echoes the mode.
- Wiki and site docs: "Queries are exact" becomes "exact below the threshold,
ANN above it, with the tail exact", with the recall and latency measured
in phase 0. The performance page gets an ANN column.
- CI: a `test/s3vectors` leg with the worker and a low threshold, gated on
recall.
### Phase 5 — Enterprise distributed builds and placement (medium)
- Drive `plan_index` / `build_index_task` from the worker job when engines
are registered; replace the Enterprise copy of `optimize_policy` with the
shared one, which merges at the limit.
- Measure four engines on 1M × 384 with ANN: latency, warm time of a merged
segment, placement epochs per merge.
- mono s3vectors legs with the index enabled, go and rust volume servers,
single and distributed, plus the hardened TLS legs.
## Risks
- **Filtered recall.** AWS's weak spot is selective filters. Pre-filtering
keeps recall but costs probes; `minimum_nprobes`/`maximum_nprobes` is the
lever, and phase 0 has to show the latency is acceptable at 1M rows.
- **Memory.** IVF_FLAT resident bytes are about the raw vectors: 1M × 384 ×
4 bytes is 1.5 GB per copy. Enterprise spreads it by `capacity_bytes`; the
single open-source engine needs that much RAM or PQ.
- **Compaction and coverage.** Settled in phase 0: Lance 10 keeps the
bitmaps through `compact_files`, and coverage stays complete.
- **Build cost on the worker.** k-means over 1M × 384 is CPU-bound; execution
concurrency stays at one and the job reports progress so the admin UI shows
it working rather than hung.
- **Placement churn after merges.** Every merge is a new epoch; too frequent
merging makes engines re-warm constantly. `MAX_DELTAS = 8` bounds it; the
phase 5 measurement decides whether to raise it.
## Out of scope
HNSW indexes, GPU builds, per-request recall parameters in the AWS API
surface, and building an index at `CreateIndex` time.
+40
View File
@@ -0,0 +1,40 @@
# SeaweedFS vector engine
`weed-vector` serves the S3 Vectors data plane through the S3 gateway. Indexes use Lance tables on the cluster's filers and volume servers. Go and Rust volume servers use the same engine and API.
With a local `weed mini` cluster running:
```sh
weed-vector --masters=localhost:9333 --metrics-port=9329
```
Point an AWS S3 Vectors client at the S3 endpoint, normally `http://localhost:8333`. The engine registers with the masters automatically. `--listen` defaults to `127.0.0.1:8090`; use an address reachable from the gateway when running on another host or in a container.
| Setting | Default | Purpose |
| --- | --- | --- |
| `--masters` | Required | Comma-separated master addresses |
| `--listen` | `127.0.0.1:8090` | gRPC listener and registered address |
| `--filer-group` | Empty | Filer group used for table storage |
| `--read-cache-mb` | `512` | Immutable data/index read cache; zero disables it |
| `--nprobes` | Automatic | ANN partitions searched per query |
| `--metrics-port` | `0` | HTTP metrics and health listener; zero disables it |
| `--metrics-ip` | `127.0.0.1` | HTTP listener address |
| `--security-toml` | Discovered | Cluster certificates, signing keys and encryption keys |
The HTTP listener provides `/health`, `/ready`, and `/metrics`. Readiness requires an active registration with the master. The gRPC listener also implements the standard health service for `vector_pb.VectorService` and the whole server. These health statuses follow master registration.
Metrics use the `SeaweedFS_vector_` prefix:
- `connected`: active master registration.
- `request_total{type,code}`, `request_seconds{type}`, and `in_flight_requests{type}`: engine requests and their gRPC outcomes.
- `query_results_total`: vectors returned by queries.
- `cache_requests_total{result}`, `cache_bytes`, and `cache_entries`: immutable read cache activity and occupancy.
- `storage_read_bytes_total` and `storage_read_seconds`: storage fetches for cacheable immutable reads.
The S3 gateway separately publishes `SeaweedFS_s3_` request metrics with `type="s3vectors.<Operation>"`. Scrape both processes to distinguish gateway latency from engine work. Metrics and HTTP probes are unauthenticated; bind them to an internal address or restrict their network access.
For mutual TLS, configure `[grpc.vector]` for the engine's server certificate, `[grpc.client]` for its master/filer connections, and the cluster CA. HTTPS volume reads also use `[https.client]`. The engine follows the same signing and encryption configuration as the other cluster clients.
Run the Lance maintenance worker against the same masters and Lance namespace to build and update ANN indexes. Queries can scan data that has not been indexed yet. See the [maintenance worker documentation](../seaweed-worker/README.md) and [Helm chart](../k8s/charts/seaweedfs/README.md) for deployment settings.
The integration suite at `test/s3vectors` runs the same API and observability checks against both Go and Rust volume servers, including mutual TLS, signed volume reads, and encrypted volume data.
@@ -13,6 +13,8 @@ pub struct IndexCoverage {
pub index_name: String,
/// Physical index segments: the IndexMetadata deltas sharing the name.
pub segments: usize,
/// IVF partitions of the largest segment.
pub partitions: usize,
/// False when any delta of this index lacks a fragment bitmap: the index
/// exists but which fragments it covers is unknown.
pub coverage_known: bool,
@@ -28,6 +30,7 @@ pub async fn coverage(dataset: &Dataset, vector_column: &str) -> Result<Vec<Inde
.field(vector_column)
.with_context(|| format!("no such column {vector_column}"))?;
let mut by_name: BTreeMap<String, Vec<Option<RoaringBitmap>>> = BTreeMap::new();
let mut partitions: BTreeMap<String, usize> = BTreeMap::new();
for index in dataset.load_indices().await?.iter() {
if index.fields != [field.id] {
continue;
@@ -36,6 +39,9 @@ pub async fn coverage(dataset: &Dataset, vector_column: &str) -> Result<Vec<Inde
.entry(index.name.clone())
.or_default()
.push(index.fragment_bitmap.clone());
let count = crate::index::segment_partitions(dataset, vector_column, &index.uuid).await?;
let most = partitions.entry(index.name.clone()).or_default();
*most = count.max(*most);
}
let all: RoaringBitmap = dataset.fragments().iter().map(|f| f.id as u32).collect();
let mut out = Vec::new();
@@ -58,6 +64,7 @@ pub async fn coverage(dataset: &Dataset, vector_column: &str) -> Result<Vec<Inde
}
}
out.push(IndexCoverage {
partitions: partitions.get(&name).copied().unwrap_or(0),
index_name: name,
segments: bitmaps.len(),
coverage_known,
+565
View File
@@ -0,0 +1,565 @@
//! Sizing the vector index an S3 Vectors table searches through, and the
//! probes a query spends on it. Building lives in the worker crate.
use anyhow::Result;
use lance::Dataset;
use lance::index::{DatasetIndexExt, DatasetIndexInternalExt};
pub use weed_lance_worker::vector_index::{
INDEX_NAME, MAX_PARTITIONS, build, distance_metric, index_names, optimize, partitions_for,
};
pub const MIN_PROBES: usize = 8;
/// How many partitions a query searches: `min` always, and under a prefilter
/// up to `max` until it has k rows.
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub struct Probes {
pub min: usize,
pub max: usize,
}
/// Probes for an index of `partitions`: a tenth of them, at least
/// [`MIN_PROBES`], never more than exist; a filtered query may widen to four
/// times that.
pub fn probes_for(partitions: usize) -> Probes {
let partitions = partitions.max(1);
let min = MIN_PROBES.max(partitions.div_ceil(10)).min(partitions);
Probes {
min,
max: (min * 4).min(partitions),
}
}
/// Partitions of one index segment, read from the opened index. Lance's index
/// statistics cost several milliseconds and may rewrite an old manifest.
pub async fn segment_partitions(
dataset: &Dataset,
vector_column: &str,
segment: &uuid::Uuid,
) -> Result<usize> {
let opened = dataset
.open_vector_index(
vector_column,
segment,
&lance_index::metrics::NoOpMetricsCollector,
)
.await?;
Ok(opened.total_partitions())
}
/// The probes a query of `vector_column` uses when it names none, or `None`
/// without an index. The count comes from the most partitions any index on
/// the column has, whichever of them the scan ends up searching; the scan
/// then finds the index it was read from already open.
pub async fn default_probes(dataset: &Dataset, vector_column: &str) -> Result<Option<Probes>> {
let Some(field) = dataset.schema().field(vector_column) else {
return Ok(None);
};
let mut partitions = None;
for index in dataset.load_indices().await?.iter() {
if index.fields != [field.id] {
continue;
}
let count = segment_partitions(dataset, vector_column, &index.uuid).await?;
partitions = Some(count.max(partitions.unwrap_or(0)));
}
Ok(partitions.map(probes_for))
}
#[cfg(test)]
mod tests {
use super::*;
use crate::coverage::coverage;
use crate::query::{self, Query};
use crate::table::VectorTable;
use crate::testing::{self, DIM};
use lance::Dataset;
use lance::dataset::optimize::{CompactionOptions, compact_files};
use lance_linalg::distance::DistanceType;
use tempfile::TempDir;
#[test]
fn probes_are_a_tenth_with_a_floor() {
assert_eq!(probes_for(1), Probes { min: 1, max: 1 });
assert_eq!(probes_for(8), Probes { min: 8, max: 8 });
assert_eq!(probes_for(64), Probes { min: 8, max: 32 });
assert_eq!(probes_for(224), Probes { min: 23, max: 92 });
assert_eq!(
probes_for(4096),
Probes {
min: 410,
max: 1640
}
);
}
fn exact(vector: &[f32], k: usize) -> Query {
Query {
vector: vector.to_vec(),
k,
filter: None,
postfilter: false,
distance: DistanceType::L2,
nprobes: None,
index_segments: None,
fragments: None,
}
}
/// The test vectors repeat, so ties are common: compare distances, not rows.
async fn assert_same_top_k(dataset: &Dataset, vector: &[f32], k: usize) {
let table = VectorTable::resolve(dataset, None, None).unwrap();
let q = exact(vector, k);
let flat = query::flat(dataset, &table, &q).await.unwrap();
let ann = query::ann(
dataset,
&table,
&Query {
nprobes: Some(MAX_PARTITIONS),
..q
},
)
.await
.unwrap();
assert_eq!(flat.len(), ann.len());
for (f, a) in flat.iter().zip(ann.iter()) {
assert!(
(f.distance - a.distance).abs() < 1e-6,
"{flat:?} vs {ann:?}"
);
}
}
/// Pseudo-random rows, so partitions are not degenerate; a different seed
/// gives vectors the table does not hold.
fn scattered(rows: usize, dim: usize, seed: u64) -> Vec<Vec<f32>> {
let mut state: u64 = seed;
(0..rows)
.map(|_| {
(0..dim)
.map(|_| {
state = state
.wrapping_mul(6364136223846793005)
.wrapping_add(1442695040888963407);
((state >> 40) as f32) / (1u64 << 24) as f32
})
.collect()
})
.collect()
}
#[tokio::test]
async fn default_probes_follow_the_index_and_find_the_neighbours() {
let dir = TempDir::new().unwrap();
let rows = scattered(4000, 8, 0x9E3779B97F4A7C15);
let tags: Vec<&str> = (0..4000)
.map(|i| if i % 100 == 0 { "rare" } else { "common" })
.collect();
let mut dataset = testing::write_dataset(&dir, &rows, &tags).await;
let table = VectorTable::resolve(&dataset, None, None).unwrap();
assert_eq!(default_probes(&dataset, "vector").await.unwrap(), None);
build(&mut dataset, &table.vector_column, INDEX_NAME, 64, false)
.await
.unwrap();
assert_eq!(
default_probes(&dataset, "vector").await.unwrap(),
Some(probes_for(64))
);
let mut found = 0;
let total = 20 * 10;
for vector in &scattered(20, 8, 0xD1B54A32D192ED03) {
let q = exact(vector, 10);
let flat: std::collections::HashSet<u64> = query::flat(&dataset, &table, &q)
.await
.unwrap()
.iter()
.map(|h| h.row_id)
.collect();
let ann = query::ann(&dataset, &table, &q).await.unwrap();
assert_eq!(ann.len(), 10);
found += ann.iter().filter(|h| flat.contains(&h.row_id)).count();
}
assert!(found * 10 >= total * 9, "recall {found}/{total}");
// Eight probes do not hold ten of the 1% rare rows; a filtered query
// widens past the minimum until it has them.
let filtered = query::ann(
&dataset,
&table,
&Query {
filter: Some("tag = 'rare'".to_string()),
..exact(&rows[0], 10)
},
)
.await
.unwrap();
assert_eq!(filtered.len(), 10);
}
#[tokio::test]
async fn default_probes_take_the_largest_index() {
use lance::index::DatasetIndexExt;
use lance::index::vector::VectorIndexParams;
use lance_index::IndexType;
let dir = TempDir::new().unwrap();
let mut dataset =
testing::write_dataset(&dir, &scattered(1000, 8, 0x9E3779B97F4A7C15), &["a"; 1000])
.await;
testing::append_dataset(
&mut dataset,
&scattered(1000, 8, 0xD1B54A32D192ED03),
&["b"; 1000],
)
.await;
for (name, fragment, partitions) in [("a-small", 0u32, 2), ("z-large", 1u32, 100)] {
dataset
.create_index_builder(
&["vector"],
IndexType::Vector,
&VectorIndexParams::ivf_flat(partitions, DistanceType::L2),
)
.name(name.to_string())
.fragments(vec![fragment])
.await
.unwrap();
dataset.checkout_latest().await.unwrap();
}
assert_eq!(
default_probes(&dataset, "vector").await.unwrap(),
Some(probes_for(100))
);
}
#[tokio::test]
async fn build_covers_every_fragment() {
let dir = TempDir::new().unwrap();
let mut dataset =
testing::write_dataset(&dir, &testing::vectors(200, DIM), &["a"; 200]).await;
testing::append_dataset(&mut dataset, &testing::vectors(50, DIM), &["b"; 50]).await;
let table = VectorTable::resolve(&dataset, None, None).unwrap();
build(
&mut dataset,
&table.vector_column,
INDEX_NAME,
partitions_for(250),
false,
)
.await
.unwrap();
let cov = coverage(&dataset, "vector").await.unwrap();
assert_eq!(cov.len(), 1);
assert_eq!(cov[0].index_name, INDEX_NAME);
assert_eq!(cov[0].segments, 1);
assert!(cov[0].coverage_known);
assert!(cov[0].unindexed_fragments.is_empty());
for vector in testing::vectors(4, DIM) {
assert_same_top_k(&dataset, &vector, 10).await;
}
}
#[tokio::test]
async fn build_uses_the_tables_metric() {
for (metric, distance) in [
("COSINE", DistanceType::Cosine),
("DOT_PRODUCT", DistanceType::Dot),
("L2", DistanceType::L2),
] {
let dir = TempDir::new().unwrap();
let rows = testing::vectors(300, DIM);
let mut dataset =
testing::write_dataset_with_metric(&dir, &rows, &["a"; 300], metric).await;
let table = VectorTable::resolve(&dataset, None, None).unwrap();
assert_eq!(distance_metric(&dataset, "vector"), distance);
build(&mut dataset, &table.vector_column, INDEX_NAME, 4, false)
.await
.unwrap();
for vector in testing::vectors(3, DIM) {
let q = Query {
distance,
..exact(&vector, 10)
};
let flat = query::flat(&dataset, &table, &q).await.unwrap();
let ann = query::ann(
&dataset,
&table,
&Query {
nprobes: Some(4),
..q
},
)
.await
.unwrap();
assert_eq!(flat.len(), ann.len(), "{metric}");
for (f, a) in flat.iter().zip(ann.iter()) {
assert!(
(f.distance - a.distance).abs() < 1e-5,
"{metric}: {flat:?} vs {ann:?}"
);
}
}
}
}
/// An index built as cosine on a table without metric metadata keeps
/// cosine through a retrain, and gets partitions for the new row count.
#[tokio::test]
async fn retrain_keeps_the_indexs_own_metric() {
use lance::index::DatasetIndexExt;
use lance::index::vector::VectorIndexParams;
use lance_index::IndexType;
use weed_lance_worker::vector_index::index_settings;
let dir = TempDir::new().unwrap();
let mut dataset =
testing::write_dataset(&dir, &testing::vectors(100, DIM), &["a"; 100]).await;
let params = VectorIndexParams::ivf_flat(2, DistanceType::Cosine);
dataset
.create_index(
&["vector"],
IndexType::Vector,
Some("idx".to_string()),
&params,
false,
)
.await
.unwrap();
dataset.checkout_latest().await.unwrap();
testing::append_dataset(&mut dataset, &testing::vectors(800, DIM), &["b"; 800]).await;
optimize(&mut dataset, "vector", true).await.unwrap();
let settings = index_settings(&dataset, "idx").await.unwrap();
assert_eq!(settings.metric, Some(DistanceType::Cosine));
assert_eq!(settings.index_type, "IVF_FLAT");
assert_eq!(settings.partitions, partitions_for(900));
}
#[tokio::test]
async fn optimize_appends_then_retrain_folds_segments() {
let dir = TempDir::new().unwrap();
let mut dataset =
testing::write_dataset(&dir, &testing::vectors(100, DIM), &["a"; 100]).await;
let table = VectorTable::resolve(&dataset, None, None).unwrap();
build(&mut dataset, &table.vector_column, INDEX_NAME, 4, false)
.await
.unwrap();
let mut appended = testing::vectors(40, DIM);
appended.push(vec![9.0; DIM]);
testing::append_dataset(&mut dataset, &appended, &["b"; 41]).await;
optimize(&mut dataset, &table.vector_column, false)
.await
.unwrap();
let cov = &coverage(&dataset, "vector").await.unwrap()[0];
assert_eq!(cov.segments, 2);
assert!(cov.unindexed_fragments.is_empty());
let hits = query::ann(
&dataset,
&table,
&Query {
nprobes: Some(4),
..exact(&[9.0; DIM], 1)
},
)
.await
.unwrap();
assert_eq!(hits[0].key, "k140", "{hits:?}");
testing::append_dataset(&mut dataset, &testing::vectors(40, DIM), &["c"; 40]).await;
optimize(&mut dataset, &table.vector_column, true)
.await
.unwrap();
let cov = &coverage(&dataset, "vector").await.unwrap()[0];
assert_eq!(cov.segments, 1);
assert!(cov.coverage_known);
assert!(cov.unindexed_fragments.is_empty());
for vector in testing::vectors(4, DIM) {
assert_same_top_k(&dataset, &vector, 10).await;
}
}
/// Append and optimize nine times: the deltas climb to the limit and then
/// fold into one, and nothing is ever uncovered after an optimize.
#[tokio::test]
async fn deltas_stay_under_the_limit_across_cycles() {
use weed_lance_worker::vector_index::{MAX_DELTAS, coverage as covered};
let dir = TempDir::new().unwrap();
let mut dataset =
testing::write_dataset(&dir, &testing::vectors(100, DIM), &["a"; 100]).await;
let table = VectorTable::resolve(&dataset, None, None).unwrap();
build(&mut dataset, &table.vector_column, INDEX_NAME, 4, false)
.await
.unwrap();
let mut seen = Vec::new();
for round in 0..9 {
testing::append_dataset(&mut dataset, &testing::vectors(20, DIM), &["b"; 20]).await;
optimize(&mut dataset, &table.vector_column, false)
.await
.unwrap();
let cover = covered(&dataset, "vector").await.unwrap().unwrap();
assert_eq!(cover.unindexed_rows, 0, "round {round}");
seen.push(cover.deltas[INDEX_NAME]);
}
assert_eq!(seen, vec![2, 3, 4, 5, 6, 7, 8, 1, 2]);
assert!(seen.iter().all(|&d| d <= MAX_DELTAS));
}
/// Two names covering disjoint fragments, as a sharded build leaves them:
/// coverage is their union, and an optimize extends one name only.
#[tokio::test]
async fn shards_stay_disjoint_through_an_optimize() {
use lance::index::DatasetIndexExt;
use lance::index::vector::VectorIndexParams;
use lance_index::IndexType;
use weed_lance_worker::vector_index::coverage as covered;
let dir = TempDir::new().unwrap();
let mut dataset =
testing::write_dataset(&dir, &testing::vectors(60, DIM), &["a"; 60]).await;
testing::append_dataset(&mut dataset, &testing::vectors(60, DIM), &["b"; 60]).await;
let params = VectorIndexParams::ivf_flat(2, DistanceType::L2);
for (name, fragment) in [("shard-0", 0u32), ("shard-1", 1u32)] {
dataset
.create_index_builder(&["vector"], IndexType::Vector, &params)
.name(name.to_string())
.fragments(vec![fragment])
.await
.unwrap();
dataset.checkout_latest().await.unwrap();
}
let cover = covered(&dataset, "vector").await.unwrap().unwrap();
assert_eq!((cover.indexed_rows, cover.unindexed_rows), (120, 0));
assert_eq!(cover.deltas.len(), 2);
testing::append_dataset(&mut dataset, &testing::vectors(30, DIM), &["c"; 30]).await;
let table = VectorTable::resolve(&dataset, None, None).unwrap();
optimize(&mut dataset, &table.vector_column, false)
.await
.unwrap();
let cover = covered(&dataset, "vector").await.unwrap().unwrap();
assert_eq!(cover.unindexed_rows, 0);
assert_eq!(
cover.deltas.values().sum::<usize>(),
3,
"{:?}",
cover.deltas
);
let per_name = coverage(&dataset, "vector").await.unwrap();
let untouched = per_name
.iter()
.filter(|c| c.segments == 1 && c.indexed_fragments.len() == 1)
.count();
assert_eq!(untouched, 1, "{per_name:?}");
for vector in testing::vectors(3, DIM) {
assert_same_top_k(&dataset, &vector, 10).await;
}
}
/// A retrain is due for one IVF_FLAT name the table outgrew, never for
/// shards sized each for their own fragments or for another index type,
/// and an explicit retrain of shards leaves them as they are.
#[tokio::test]
async fn retrain_is_due_for_one_ivf_flat_name_only() {
use lance::index::DatasetIndexExt;
use lance::index::vector::VectorIndexParams;
use lance_index::IndexType;
use weed_lance_worker::vector_index::{coverage as covered, retrain_due};
let grown = |dir: &TempDir| {
let path = dir.path().to_str().unwrap().to_string();
async move {
let mut dataset =
testing::write_dataset_at(&path, &testing::vectors(300, DIM), &["a"; 300])
.await;
testing::append_dataset(&mut dataset, &testing::vectors(300, DIM), &["b"; 300])
.await;
dataset
}
};
let dir = TempDir::new().unwrap();
let mut dataset = grown(&dir).await;
build(&mut dataset, "vector", INDEX_NAME, 2, false)
.await
.unwrap();
assert!(retrain_due(&dataset, "vector").await.unwrap());
let dir = TempDir::new().unwrap();
let mut dataset = grown(&dir).await;
let params = VectorIndexParams::ivf_pq(2, 8, 2, DistanceType::L2, 10);
dataset
.create_index(&["vector"], IndexType::Vector, None, &params, false)
.await
.unwrap();
dataset.checkout_latest().await.unwrap();
assert!(!retrain_due(&dataset, "vector").await.unwrap());
let dir = TempDir::new().unwrap();
let mut dataset = grown(&dir).await;
let params = VectorIndexParams::ivf_flat(2, DistanceType::L2);
for (name, fragment) in [("shard-0", 0u32), ("shard-1", 1u32)] {
dataset
.create_index_builder(&["vector"], IndexType::Vector, &params)
.name(name.to_string())
.fragments(vec![fragment])
.await
.unwrap();
dataset.checkout_latest().await.unwrap();
}
assert!(!retrain_due(&dataset, "vector").await.unwrap());
let before = covered(&dataset, "vector").await.unwrap();
assert!(optimize(&mut dataset, "vector", true).await.is_err());
assert_eq!(covered(&dataset, "vector").await.unwrap(), before);
let per_name = coverage(&dataset, "vector").await.unwrap();
assert!(
per_name.iter().all(|c| c.indexed_fragments.len() == 1),
"{per_name:?}"
);
}
#[tokio::test]
async fn compaction_keeps_the_index_covering() {
let dir = TempDir::new().unwrap();
let mut dataset =
testing::write_dataset(&dir, &testing::vectors(60, DIM), &["a"; 60]).await;
for tag in ["b", "c", "d"] {
testing::append_dataset(&mut dataset, &testing::vectors(60, DIM), &[tag; 60]).await;
}
let table = VectorTable::resolve(&dataset, None, None).unwrap();
build(&mut dataset, &table.vector_column, INDEX_NAME, 4, false)
.await
.unwrap();
dataset.delete("key = 'k0'").await.unwrap();
assert_eq!(dataset.get_fragments().len(), 4);
let options = CompactionOptions {
target_rows_per_fragment: 1000,
materialize_deletions: true,
materialize_deletions_threshold: 0.0,
..Default::default()
};
compact_files(&mut dataset, options, None).await.unwrap();
dataset.checkout_latest().await.unwrap();
assert_eq!(dataset.get_fragments().len(), 1);
let cov = &coverage(&dataset, "vector").await.unwrap()[0];
assert!(cov.coverage_known);
assert!(cov.unindexed_fragments.is_empty(), "{cov:?}");
let hits = query::ann(
&dataset,
&table,
&Query {
nprobes: Some(MAX_PARTITIONS),
..exact(&testing::vectors(1, DIM)[0], 10)
},
)
.await
.unwrap();
let mut tied: Vec<&str> = hits
.iter()
.filter(|h| h.distance < 1e-6)
.map(|h| h.key.as_str())
.collect();
tied.sort();
assert_eq!(tied, ["k120", "k180", "k60"], "{hits:?}");
for vector in testing::vectors(4, DIM) {
assert_same_top_k(&dataset, &vector, 10).await;
}
}
}
+1
View File
@@ -7,6 +7,7 @@
pub mod coverage;
mod filter;
pub mod index;
pub mod maintenance;
pub mod query;
pub mod segments;
+200 -31
View File
@@ -1,17 +1,21 @@
use anyhow::{Context, Result, bail};
use clap::{Parser, Subcommand};
use weed_lance_worker::catalog::{NamespaceClient, parse_id};
use weed_lance_worker::dataset::{self, FallbackOptions};
use weed_lance_worker::catalog::{NamespaceClient, SigningKey, TableDescription, parse_id};
use weed_lance_worker::dataset::{self, Storage};
use weed_vector_core::coverage;
use weed_vector_core::index;
use weed_vector_core::query::{self, Query};
use weed_vector_core::table::VectorTable;
/// Mirrors `weed-worker`'s flags where the two touch the same thing: the
/// namespace is WEED_LANCE_NAMESPACE and the fallback storage credentials are
/// the same env vars.
/// namespace that says where a table is, and the masters whose filers hold
/// it.
#[derive(Parser, Debug)]
#[command(name = "weed-vector", about = "SeaweedFS vector query spike")]
#[command(
name = "weed-vector",
about = "Describe, query and index S3 Vectors tables"
)]
struct Args {
/// Lance namespace the tables live in.
#[arg(
@@ -21,12 +25,29 @@ struct Args {
)]
namespace: String,
/// Storage credentials to use where the namespace vends none.
#[arg(long, env = "WEED_S3_ACCESS_KEY")]
access_key: Option<String>,
/// Comma-separated master addresses, whose filers the tables are kept
/// through. A table named by a local URI needs none.
#[arg(long, alias = "master", env = "WEED_MASTERS")]
masters: Option<String>,
#[arg(long, env = "WEED_S3_SECRET_KEY")]
secret_key: Option<String>,
/// The filer group whose filers hold the tables.
#[arg(
long,
alias = "filer.group",
env = "WEED_FILER_GROUP",
default_value = ""
)]
filer_group: String,
/// The `jwt.filer_signing.key` of the cluster, which the namespace
/// authenticates cluster processes with; security.toml's by default.
#[arg(long, env = "WEED_FILER_SIGNING_KEY")]
filer_signing_key: Option<String>,
/// The cluster's security.toml; by default the file weed processes find
/// in -config_dir, ., ~/.seaweedfs and /etc/seaweedfs.
#[arg(long, env = "WEED_VECTOR_SECURITY_TOML")]
security_toml: Option<std::path::PathBuf>,
#[command(subcommand)]
command: Command,
@@ -91,24 +112,67 @@ enum Command {
#[arg(long)]
key_column: Option<String>,
},
/// Build or extend the vector index, then print its coverage.
Index {
#[command(subcommand)]
action: IndexAction,
},
}
/// `id` is a delimiter-joined table id; when it looks like a URI (`s3://...`,
/// `/path/...`, `file://...`) the dataset is opened directly, skipping the
/// namespace — useful for a spike run against a local dataset.
#[derive(Subcommand, Debug)]
enum IndexAction {
/// An IVF_FLAT index in the table's metric; partitions default to the
/// square root of the row count.
Build {
id: String,
#[arg(long)]
partitions: Option<usize>,
#[arg(long, default_value = index::INDEX_NAME)]
name: String,
/// Rebuild an index of that name instead of failing.
#[arg(long)]
replace: bool,
#[arg(long)]
vector_column: Option<String>,
},
/// Cover the rows written since the index was built.
Optimize {
id: String,
/// Train the index again on every row instead of adding a segment.
#[arg(long)]
retrain: bool,
#[arg(long)]
vector_column: Option<String>,
},
}
/// `id` is a delimiter-joined table id; when it looks like a URI the dataset
/// is opened directly, skipping the namespace: an `s3://` URI on the
/// cluster's storage, a local path (`/path/...`, `file://...`) as it is.
async fn open_dataset(
client: &NamespaceClient,
id: &str,
fallback: &FallbackOptions,
storage: &Storage,
) -> Result<lance::Dataset> {
if id.contains("://") || id.starts_with('/') {
let mut builder = lance::dataset::builder::DatasetBuilder::from_uri(id);
if !fallback.is_empty() {
builder = builder.with_storage_options(fallback.clone());
}
return builder.load().await.with_context(|| format!("open {id}"));
let on_cluster = id.starts_with("s3://");
if !on_cluster && (id.contains("://") || id.starts_with('/')) {
return lance::dataset::builder::DatasetBuilder::from_uri(id)
.load()
.await
.with_context(|| format!("open {id}"));
}
let table = dataset::open(client, &parse_id(id), fallback).await?;
if storage.0.is_none() {
bail!("{id} is kept on the cluster and needs --masters");
}
if on_cluster {
let description = TableDescription {
location: id.to_string(),
storage_options: Default::default(),
managed_versioning: false,
};
return dataset::open_at(&description, storage.wrapper(id).await?, None).await;
}
let table = dataset::open(client, &parse_id(id), storage).await?;
Ok(table.dataset)
}
@@ -116,6 +180,7 @@ async fn open_dataset(
async fn main() -> Result<()> {
weed_lance_worker::tls::install_default_crypto_provider();
tracing_subscriber::fmt()
.with_writer(std::io::stderr)
.with_env_filter(
tracing_subscriber::EnvFilter::try_from_default_env()
.unwrap_or_else(|_| tracing_subscriber::EnvFilter::new("info")),
@@ -123,12 +188,21 @@ async fn main() -> Result<()> {
.init();
let args = Args::parse();
let mut fallback = FallbackOptions::new();
if let (Some(access), Some(secret)) = (args.access_key, args.secret_key) {
fallback.insert("aws_access_key_id".to_string(), access);
fallback.insert("aws_secret_access_key".to_string(), secret);
}
let client = NamespaceClient::new(args.namespace).with_fallback(&fallback);
let security =
weed_lance_worker::security::Security::load(args.security_toml.as_deref(), None)?;
let signing_key = SigningKey::new(
args.filer_signing_key
.or_else(|| security.get("jwt.filer_signing.key")),
);
let client = NamespaceClient::new(args.namespace).with_signing_key(signing_key.as_ref());
let storage = match &args.masters {
Some(masters) => {
let cluster =
weed_lance_worker::cluster::Cluster::new(masters, &args.filer_group, &security)?;
Storage(Some(cluster.storage(&security).await?))
}
None => Storage::default(),
};
match args.command {
Command::Describe {
@@ -136,7 +210,7 @@ async fn main() -> Result<()> {
vector_column,
key_column,
} => {
let dataset = open_dataset(&client, &id, &fallback).await?;
let dataset = open_dataset(&client, &id, &storage).await?;
let schema: arrow_schema::Schema = dataset.schema().into();
let resolved =
VectorTable::resolve(&dataset, vector_column.as_deref(), key_column.as_deref());
@@ -176,7 +250,7 @@ async fn main() -> Result<()> {
vector_column,
key_column,
} => {
let dataset = open_dataset(&client, &id, &fallback).await?;
let dataset = open_dataset(&client, &id, &storage).await?;
let vector =
VectorTable::resolve(&dataset, vector_column.as_deref(), key_column.as_deref())?;
let coverages = coverage::coverage(&dataset, &vector.vector_column).await?;
@@ -188,7 +262,7 @@ async fn main() -> Result<()> {
vector_column,
key_column,
} => {
let dataset = open_dataset(&client, &id, &fallback).await?;
let dataset = open_dataset(&client, &id, &storage).await?;
let vector =
VectorTable::resolve(&dataset, vector_column.as_deref(), key_column.as_deref())?;
for batch in query::get(&dataset, &vector, &keys, None).await? {
@@ -208,7 +282,7 @@ async fn main() -> Result<()> {
key_column,
nprobes,
} => {
let opened = open_dataset(&client, &id, &fallback).await?;
let opened = open_dataset(&client, &id, &storage).await?;
let dataset = match version {
Some(v) => opened.checkout_version(v).await?,
None => opened,
@@ -243,6 +317,49 @@ async fn main() -> Result<()> {
};
println!("{}", serde_json::to_string_pretty(&hits)?);
}
Command::Index { action } => {
let (id, vector_column) = match &action {
IndexAction::Build {
id, vector_column, ..
}
| IndexAction::Optimize {
id, vector_column, ..
} => (id.clone(), vector_column.clone()),
};
let mut dataset = open_dataset(&client, &id, &storage).await?;
let table = VectorTable::resolve(&dataset, vector_column.as_deref(), None)?;
let started = std::time::Instant::now();
match action {
IndexAction::Build {
partitions,
name,
replace,
..
} => {
let rows = dataset.count_rows(None).await?;
let partitions = partitions.unwrap_or_else(|| index::partitions_for(rows));
index::build(
&mut dataset,
&table.vector_column,
&name,
partitions,
replace,
)
.await?;
}
IndexAction::Optimize { retrain, .. } => {
index::optimize(&mut dataset, &table.vector_column, retrain).await?;
}
}
println!(
"{}",
serde_json::json!({
"version": dataset.version().version,
"seconds": started.elapsed().as_secs_f64(),
"coverage": coverage::coverage(&dataset, &table.vector_column).await?,
})
);
}
}
Ok(())
}
@@ -272,6 +389,58 @@ mod tests {
}
}
#[test]
fn cluster_flags_mirror_the_engine_and_the_worker() {
let args = Args::try_parse_from([
"weed-vector",
"--masters",
"m:9333",
"--filer-group",
"eu",
"describe",
"t",
])
.unwrap();
assert_eq!(args.masters.as_deref(), Some("m:9333"));
assert_eq!(args.filer_group, "eu");
assert!(
Args::try_parse_from(["weed-vector", "--encryptVolumeData", "describe", "t"]).is_err(),
"volume encryption follows the filer"
);
}
#[test]
fn index_build_takes_partitions_and_a_name() {
let args = Args::try_parse_from([
"weed-vector",
"index",
"build",
"b$vector$i",
"--partitions",
"32",
"--replace",
])
.unwrap();
match args.command {
Command::Index {
action:
IndexAction::Build {
id,
partitions,
name,
replace,
..
},
} => {
assert_eq!(id, "b$vector$i");
assert_eq!(partitions, Some(32));
assert_eq!(name, index::INDEX_NAME);
assert!(replace);
}
_ => panic!("expected index build"),
}
}
#[test]
fn column_flags_on_get_describe_coverage() {
for argv in [
+237 -5
View File
@@ -2,8 +2,8 @@
//! and version tags act as the lease maintenance must respect.
use std::collections::{HashMap, HashSet};
use std::sync::{LazyLock, Mutex};
use std::time::{Duration, SystemTime, UNIX_EPOCH};
use std::sync::{Arc, LazyLock, Mutex, Weak};
use std::time::{Duration, Instant, SystemTime, UNIX_EPOCH};
use anyhow::Result;
use lance::Dataset;
@@ -21,6 +21,11 @@ const FOREIGN_TTL: Duration = Duration::from_secs(24 * 3600);
/// Expired-lease scans run at most once per interval per dataset, not per
/// request.
const REAP_INTERVAL: Duration = Duration::from_secs(60);
/// How long a shared lease stays after the last read that took it.
const SHARED_LINGER: Duration = Duration::from_secs(10);
/// A shared lease is handed out for this long after it was taken, then
/// replaced, so a busy version's tag never ages into a reaper's TTL.
const SHARED_FOR: Duration = Duration::from_secs(300);
/// A process-unique owner id embedded in lease tags so a reaper only applies
/// the short TTL to its own leases.
static INSTANCE: LazyLock<String> = LazyLock::new(|| Uuid::new_v4().simple().to_string());
@@ -138,6 +143,106 @@ fn untrack(tag: &str) {
}
}
/// Leases that reads of one version share: the first read tags the version,
/// reads that follow reuse the tag, and the tag goes once no read has taken it
/// for [`SHARED_LINGER`]. Spares a tag write and delete per read.
pub struct SharedLeases {
/// By location, version, manifest ETag and commit time: a table dropped
/// and created again at its location reuses version numbers, not
/// manifests. A rename commit leaves no ETag.
held: Mutex<Held>,
linger: Duration,
share_for: Duration,
}
type LeaseKey = (String, u64, Option<String>, u128);
#[derive(Default)]
struct Held {
slots: HashMap<LeaseKey, Slot>,
/// Slots are pruned once they reach this many, so a lookup does not
/// scan them all.
prune_at: usize,
}
/// Reads of a version that find no lease wait here for the first to take it.
type Slot = Arc<tokio::sync::Mutex<Weak<SharedLease>>>;
impl Default for SharedLeases {
fn default() -> Self {
Self {
held: Mutex::default(),
linger: SHARED_LINGER,
share_for: SHARED_FOR,
}
}
}
/// A lease held for as long as any read, or the linger after the last one,
/// keeps it.
pub struct SharedLease {
_lease: Lease,
taken: Instant,
}
impl SharedLeases {
/// [`try_pin`], reusing a lease another read of this version holds.
pub async fn try_pin(&self, dataset: &Dataset) -> Result<Option<Arc<SharedLease>>> {
let key = (
dataset.uri().to_string(),
dataset.version().version,
dataset.manifest_location().e_tag.clone(),
dataset.manifest().timestamp_nanos,
);
let slot = match self.held.lock() {
Ok(mut held) => {
if held.slots.len() >= held.prune_at {
// A slot another read has taken but not yet locked stays.
held.slots.retain(|_, slot| {
Arc::strong_count(slot) > 1
|| slot
.try_lock()
.map_or(true, |lease| lease.strong_count() > 0)
});
held.prune_at = (2 * held.slots.len()).max(64);
// A burst's allocation would otherwise make every later
// prune scan it.
let capacity = held.prune_at;
held.slots.shrink_to(capacity);
}
held.slots.entry(key).or_default().clone()
}
Err(_) => Slot::default(),
};
let mut slot = slot.lock().await;
let lease = match slot
.upgrade()
.filter(|lease| lease.taken.elapsed() < self.share_for)
{
Some(lease) => lease,
None => {
let Some(lease) = try_pin(dataset).await? else {
return Ok(None);
};
let lease = Arc::new(SharedLease {
_lease: lease,
taken: Instant::now(),
});
*slot = Arc::downgrade(&lease);
lease
}
};
drop(slot);
if let Ok(handle) = tokio::runtime::Handle::try_current() {
let (lingering, linger) = (lease.clone(), self.linger);
handle.spawn(async move {
tokio::time::sleep(linger).await;
drop(lingering);
});
}
Ok(Some(lease))
}
}
/// Drop lease tags whose owner never released them, so an abandoned lease
/// cannot keep a version out of cleanup forever.
async fn reap_expired(dataset: &Dataset, now: u128) {
@@ -178,8 +283,7 @@ async fn reap_expired(dataset: &Dataset, now: u128) {
}
})
.is_some_and(|(own, stamp)| {
now.saturating_sub(stamp)
> if own { LEASE_TTL } else { FOREIGN_TTL }.as_nanos()
now.saturating_sub(stamp) > if own { LEASE_TTL } else { FOREIGN_TTL }.as_nanos()
});
if expired && !held {
let _ = dataset.tags().delete(name).await;
@@ -325,6 +429,131 @@ mod tests {
b.release().await.unwrap();
}
/// Reads of one version share a tag, another version gets its own, and
/// the tags go once the linger after the last read has passed.
#[tokio::test]
async fn reads_of_a_version_share_one_lease() {
let dir = TempDir::new().unwrap();
let mut dataset =
testing::write_dataset(&dir, &testing::vectors(4, 4), &["a", "b", "c", "d"]).await;
let leases = SharedLeases {
linger: Duration::from_millis(50),
..Default::default()
};
let a = leases.try_pin(&dataset).await.unwrap().unwrap();
let b = leases.try_pin(&dataset).await.unwrap().unwrap();
assert!(Arc::ptr_eq(&a, &b));
assert_eq!(dataset.tags().list().await.unwrap().len(), 1);
testing::append_dataset(&mut dataset, &testing::vectors(1, 4), &["e"]).await;
let c = leases.try_pin(&dataset).await.unwrap().unwrap();
assert!(!Arc::ptr_eq(&a, &c));
assert_eq!(dataset.tags().list().await.unwrap().len(), 2);
drop((a, b, c));
tokio::time::sleep(leases.linger).await;
let mut left = usize::MAX;
for _ in 0..1000 {
// A listing that meets a tag as it is deleted fails; try again.
if let Ok(tags) = dataset.tags().list().await {
left = tags.len();
}
if left == 0 {
break;
}
tokio::time::sleep(Duration::from_millis(10)).await;
}
assert_eq!(left, 0);
}
/// Reads that arrive together, before any lease exists, share the one the
/// first takes.
#[tokio::test]
async fn concurrent_reads_take_one_lease() {
let dir = TempDir::new().unwrap();
let dataset =
testing::write_dataset(&dir, &testing::vectors(4, 4), &["a", "b", "c", "d"]).await;
let leases = SharedLeases::default();
let pins: Vec<_> = futures::future::join_all((0..8).map(|_| leases.try_pin(&dataset)))
.await
.into_iter()
.map(|lease| lease.unwrap().unwrap())
.collect();
assert!(pins.iter().all(|lease| Arc::ptr_eq(lease, &pins[0])));
assert_eq!(dataset.tags().list().await.unwrap().len(), 1);
}
/// A table dropped and written again at its location gets its own lease
/// for a version number the old table had.
#[tokio::test]
async fn a_recreated_table_takes_its_own_lease() {
let dir = TempDir::new().unwrap();
let rows = testing::vectors(4, 4);
let old = testing::write_dataset(&dir, &rows, &["a", "b", "c", "d"]).await;
let leases = SharedLeases::default();
let first = leases.try_pin(&old).await.unwrap().unwrap();
std::fs::remove_dir_all(dir.path()).unwrap();
let new = testing::write_dataset(&dir, &rows, &["e", "f", "g", "h"]).await;
assert_eq!(new.version().version, old.version().version);
let second = leases.try_pin(&new).await.unwrap().unwrap();
assert!(!Arc::ptr_eq(&first, &second));
assert_eq!(new.tags().list().await.unwrap().len(), 1);
}
/// Slots of released leases are pruned as new versions come in.
#[tokio::test]
async fn released_lease_slots_are_pruned() {
let dir = TempDir::new().unwrap();
let mut dataset = testing::write_dataset(&dir, &testing::vectors(1, 4), &["a"]).await;
let leases = SharedLeases {
linger: Duration::ZERO,
..Default::default()
};
for _ in 0..200 {
drop(leases.try_pin(&dataset).await.unwrap());
tokio::task::yield_now().await;
testing::append_dataset(&mut dataset, &testing::vectors(1, 4), &["b"]).await;
}
assert!(leases.held.lock().unwrap().slots.len() <= 128);
}
/// A slot a read has taken but not yet locked survives a prune, so that
/// read and the next share the lease.
#[tokio::test]
async fn a_taken_slot_survives_a_prune() {
let dir = TempDir::new().unwrap();
let dataset = testing::write_dataset(&dir, &testing::vectors(1, 4), &["a"]).await;
let leases = SharedLeases::default();
let key: LeaseKey = ("elsewhere".into(), 1, None, 0);
let taken = {
let mut held = leases.held.lock().unwrap();
held.prune_at = 0;
held.slots.entry(key.clone()).or_default().clone()
};
leases.try_pin(&dataset).await.unwrap().unwrap();
assert!(Arc::ptr_eq(
&leases.held.lock().unwrap().slots[&key],
&taken
));
}
/// A shared lease is not handed out past SHARED_FOR, so a busy version
/// moves to a fresh tag.
#[tokio::test]
async fn a_shared_lease_is_replaced_when_old() {
let dir = TempDir::new().unwrap();
let dataset =
testing::write_dataset(&dir, &testing::vectors(4, 4), &["a", "b", "c", "d"]).await;
let leases = SharedLeases {
share_for: Duration::from_millis(50),
..Default::default()
};
let a = leases.try_pin(&dataset).await.unwrap().unwrap();
tokio::time::sleep(leases.share_for).await;
let b = leases.try_pin(&dataset).await.unwrap().unwrap();
assert!(!Arc::ptr_eq(&a, &b));
}
/// A stale lease tag left behind is reaped by the next pin; a stale-named
/// tag this process still holds is not.
#[tokio::test]
@@ -379,7 +608,10 @@ mod tests {
let foreign = format!("{QUERY_TAG_PREFIX}other-{stale}-foreign");
dataset
.tags()
.create(&foreign, Ref::Version(None, Some(dataset.version().version)))
.create(
&foreign,
Ref::Version(None, Some(dataset.version().version)),
)
.await
.unwrap();
let lease = pin(&dataset).await.unwrap();
+22 -2
View File
@@ -120,8 +120,28 @@ async fn scanner(
crate::filter::apply(&mut scanner, filter)?;
scanner.prefilter(!query.postfilter);
}
if let Some(nprobes) = query.nprobes {
scanner.nprobes(nprobes);
// An explicitly empty segment list searches no index, so there is
// nothing to size probes for.
let no_segments = query.index_segments.as_ref().is_some_and(|s| s.is_empty());
if use_index && !no_segments {
let probes = match query.nprobes {
Some(n) => Some(crate::index::Probes { min: n, max: n }),
None => crate::index::default_probes(dataset, &table.vector_column).await?,
};
if let Some(probes) = probes {
// Lance searches `min` partitions, and under a prefilter keeps
// going up to `max` until it has k rows. An open bound would
// make it prepare every partition for that, which costs as much
// as the exact scan it is meant to avoid.
scanner.minimum_nprobes(probes.min);
scanner.maximum_nprobes(probes.max);
tracing::debug!(
min = probes.min,
max = probes.max,
filtered = query.filter.is_some(),
"ann scan"
);
}
}
let segments = query.index_segments.as_deref().unwrap_or(&[]);
if use_index && !segments.is_empty() {
+34
View File
@@ -66,6 +66,40 @@ pub async fn write_dataset_at(uri: &str, vectors: &[Vec<f32>], tags: &[&str]) ->
Dataset::write(reader, uri, None).await.unwrap()
}
/// Like [`write_dataset`], with the vector field carrying the `distance_metric`
/// metadata `CreateIndex` writes (`L2`, `COSINE`, `DOT_PRODUCT`).
pub async fn write_dataset_with_metric(
dir: &TempDir,
vectors: &[Vec<f32>],
tags: &[&str],
metric: &str,
) -> Dataset {
let dim = vectors[0].len();
let plain = schema(dim);
let fields: Vec<Arc<Field>> =
plain
.fields()
.iter()
.map(|f| {
if f.name() == "vector" {
Arc::new(f.as_ref().clone().with_metadata(
[("distance_metric".to_string(), metric.to_string())].into(),
))
} else {
f.clone()
}
})
.collect();
let schema = Arc::new(Schema::new(fields));
let batch = batch(0, vectors, tags, dim)
.with_schema(schema.clone())
.unwrap();
let reader = RecordBatchIterator::new(vec![Ok(batch)].into_iter(), schema);
Dataset::write(reader, dir.path().to_str().unwrap(), None)
.await
.unwrap()
}
/// Deterministic non-trivial vectors for index coverage tests.
pub fn vectors(rows: usize, dim: usize) -> Vec<Vec<f32>> {
(0..rows)
+6 -1
View File
@@ -13,8 +13,12 @@ name = "weed-vector"
path = "src/main.rs"
[dependencies]
axum = "0.8"
prometheus = { version = "0.13", default-features = false }
tonic-health = "0.14"
weed-vector-core = { path = "../core" }
weed-lance-worker = { path = "../../../seaweed-worker/crates/lance" }
sw-client = { path = "../../../seaweed-worker/crates/sw-client" }
lance = { version = "10", default-features = false, features = ["aws"] }
lance-arrow = "10"
lance-linalg = "10"
@@ -32,7 +36,8 @@ bytes = "1"
clap = { version = "4", features = ["derive", "env"] }
futures = "0.3"
moka = { version = "0.12", features = ["sync"] }
object_store = "0.13"
uuid = { version = "1", features = ["v4"] }
object_store = { version = "0.13", features = ["aws"] }
serde = { version = "1", features = ["derive"] }
serde_json = "1"
tokio.workspace = true
+8
View File
@@ -1,5 +1,7 @@
//! The VectorService gRPC server the S3 gateway talks to.
pub mod metrics;
pub mod observed;
pub mod read_cache;
pub mod service;
@@ -13,6 +15,12 @@ pub mod master_pb {
tonic::include_proto!("master_pb");
}
pub mod master_client;
pub mod security {
pub use weed_lance_worker::security::*;
}
pub mod tls {
pub use weed_lance_worker::grpc_tls::*;
}
pub mod pb {
pub mod vector {
+199 -55
View File
@@ -1,47 +1,83 @@
use std::path::PathBuf;
use anyhow::{Context, Result};
use clap::Parser;
use tokio_stream::wrappers::TcpListenerStream;
use tonic::transport::Server;
use tonic::transport::server::TcpIncoming;
use weed_lance_worker::catalog::{NamespaceClient, SigningKey};
use weed_lance_worker::dataset::FallbackOptions;
use weed_lance_worker::cluster::Cluster;
use weed_vector_server::pb::vector::vector_service_server::VectorServiceServer;
use weed_vector_server::read_cache::ReadCache;
use weed_vector_server::security::Security;
use weed_vector_server::service::VectorServer;
use weed_vector_server::tls::{self, ServerTls};
use weed_vector_server::{
master_client,
metrics::{self, Metrics},
observed::ObservedService,
};
/// The VectorService the S3 gateway fans vector calls out to.
#[derive(Parser)]
struct Args {
/// Lance Namespace base URL: the S3 gateway's Lance port (-s3.port.lance).
#[arg(
long,
env = "WEED_VECTOR_NAMESPACE",
default_value = "http://localhost:9101"
)]
namespace: String,
/// gRPC listen address.
#[arg(long, env = "WEED_VECTOR_LISTEN", default_value = "127.0.0.1:8090")]
listen: String,
/// object_store option fallback for deployments that vend no credentials:
/// key=value, repeatable.
#[arg(long = "storage-option", alias = "storage.option", value_parser = parse_option)]
storage_options: Vec<(String, String)>,
/// Comma-separated master addresses to register this server with; the
/// gateway discovers it via ListClusterNodes(client_type=vector).
/// Address to register when it differs from the listen address.
#[arg(long, env = "WEED_VECTOR_ADVERTISE")]
advertise: Option<String>,
/// Comma-separated master addresses. The engine registers with them, so
/// the gateway finds it, and asks them for the filers it keeps tables
/// through.
#[arg(long, alias = "master", env = "WEED_VECTOR_MASTERS")]
masters: Option<String>,
masters: String,
/// The filer group whose filers the engine keeps tables through.
#[arg(
long,
alias = "filer.group",
env = "WEED_VECTOR_FILER_GROUP",
default_value = ""
)]
filer_group: String,
/// Memory for caching reads of write-once table files, in MiB; 0 disables.
#[arg(long, env = "WEED_VECTOR_READ_CACHE_MB", default_value_t = 512)]
read_cache_mb: u64,
/// The `jwt.filer_signing.key` of the cluster. The Lance namespace vends
/// credentials that write index data only to a caller holding it.
#[arg(long, env = "WEED_FILER_SIGNING_KEY")]
filer_signing_key: Option<String>,
}
/// Memory for indexes loaded by queries, kept for the next query, in MiB.
#[arg(long, env = "WEED_VECTOR_INDEX_CACHE_MB", default_value_t = 1024)]
index_cache_mb: usize,
/// Memory for table metadata (manifests, file footers) kept across
/// queries, in MiB.
#[arg(long, env = "WEED_VECTOR_METADATA_CACHE_MB", default_value_t = 256)]
metadata_cache_mb: usize,
/// Partitions each query probes in a table's vector index. Default: a
/// tenth of the index's partitions, at least 8; more under a filter.
#[arg(long, env = "WEED_VECTOR_NPROBES")]
nprobes: Option<std::num::NonZeroUsize>,
/// The cluster's security.toml: [grpc.vector] for this server,
/// [grpc.client] for the masters and filers, https.client for volume
/// servers under HTTPS, jwt.signing.read.key for signed volume reads,
/// s3.sse.kek or s3.sse.key for SSE-S3 buckets. By default the file weed
/// processes find in -config_dir, ., ~/.seaweedfs and /etc/seaweedfs.
#[arg(long, env = "WEED_VECTOR_SECURITY_TOML")]
security_toml: Option<PathBuf>,
#[arg(
long,
alias = "metricsPort",
env = "WEED_VECTOR_METRICS_PORT",
default_value_t = 0
)]
metrics_port: u16,
#[arg(
long,
alias = "metricsIp",
env = "WEED_VECTOR_METRICS_IP",
default_value = "127.0.0.1"
)]
metrics_ip: String,
fn parse_option(s: &str) -> Result<(String, String), String> {
s.split_once('=')
.map(|(k, v)| (k.to_string(), v.to_string()))
.ok_or_else(|| "expected key=value".to_string())
/// The directory weed's -config_dir names, searched for security.toml
/// first.
#[arg(long, alias = "config_dir", env = "WEED_VECTOR_CONFIG_DIR")]
config_dir: Option<PathBuf>,
}
// weed tools take Go-style single-dash long flags; normalize_args converts
@@ -71,7 +107,13 @@ fn normalize_args(args: Vec<String>) -> Vec<String> {
/// A wildcard listen address is not reachable for peers; derive the outbound
/// IP toward the first master instead — the address the master sees anyway.
fn advertised_addr(listen: &str, masters: &[String]) -> String {
fn advertised_addr(listen: &str, masters: &[String], advertise: Option<&str>) -> String {
if let Some(address) = advertise
.map(str::trim)
.filter(|address| !address.is_empty())
{
return address.to_string();
}
let Some((host, _)) = listen.rsplit_once(':') else {
return listen.to_string();
};
@@ -105,40 +147,142 @@ fn advertised_addr(listen: &str, masters: &[String]) -> String {
async fn main() -> Result<()> {
tracing_subscriber::fmt::init();
let args = Args::parse_from(normalize_args(std::env::args().collect()));
weed_lance_worker::tls::install_default_crypto_provider();
let security = Security::load(args.security_toml.as_deref(), args.config_dir.as_deref())?;
if let Some(path) = security.path() {
tracing::info!("security settings from {}", path.display());
}
let server_tls = security
.grpc_server_tls("grpc.vector")?
.map(ServerTls::load)
.transpose()?;
// Bind before registering, so the master never lists an engine that is
// about to exit because its port is taken.
let incoming = TcpIncoming::bind(args.listen.parse()?)
.with_context(|| format!("listen on {}", args.listen))?
.with_nodelay(Some(true));
if let Some(masters) = &args.masters {
let masters: Vec<String> = masters.split(',').map(|s| s.trim().to_string()).collect();
for master in &masters {
weed_vector_server::master_client::grpc_addr(master)?;
}
let advertise = advertised_addr(&args.listen, &masters);
tokio::spawn(weed_vector_server::master_client::keep_connected(
masters, advertise,
));
let listener = tokio::net::TcpListener::bind(&args.listen)
.await
.with_context(|| format!("listen on {}", args.listen))?;
let cluster = Cluster::new(&args.masters, &args.filer_group, &security)?;
let reads = cluster.storage(&security).await?;
let (health, health_service) = tonic_health::server::health_reporter();
let metrics = Metrics::new()?.with_health(health);
metrics.set_connected(false).await;
let cache = (args.read_cache_mb > 0)
.then(|| ReadCache::new(args.read_cache_mb << 20).with_metrics(metrics.clone()));
let mut service = VectorServer::new()
.with_nprobes(args.nprobes.map(std::num::NonZeroUsize::get))
.with_lance_caches(args.index_cache_mb << 20, args.metadata_cache_mb << 20)
.with_storage(reads);
if let Some(cache) = &cache {
service = service.with_read_cache(cache.clone());
}
let fallback: FallbackOptions = args.storage_options.into_iter().collect();
let signing_key = SigningKey::new(args.filer_signing_key);
let client = NamespaceClient::new(args.namespace.clone())
.with_fallback(&fallback)
.with_signing_key(signing_key.as_ref());
let mut service = VectorServer::new(client, fallback);
if args.read_cache_mb > 0 {
service = service.with_read_cache(ReadCache::new(args.read_cache_mb << 20));
if args.metrics_port > 0 {
let ip = args
.metrics_ip
.trim_start_matches('[')
.trim_end_matches(']')
.parse::<std::net::IpAddr>()?;
let listener =
tokio::net::TcpListener::bind(std::net::SocketAddr::new(ip, args.metrics_port)).await?;
let metrics = metrics.clone();
tokio::spawn(async move {
if let Err(error) = metrics::serve_listener(metrics, listener, cache).await {
tracing::error!(%error, "vector metrics server stopped");
}
});
}
tracing::info!(namespace = %args.namespace, listen = %args.listen, "weed-vector");
Server::builder()
tracing::info!(
listen = %args.listen,
"weed-vector; serving gRPC {}, dialing masters {}",
match &server_tls {
Some(tls) if tls.files().allowed_common_names.is_empty()
&& tls.files().allowed_wildcard_domain.is_empty() =>
"over TLS with client certificates required",
Some(_) => "over TLS with allowed client certificates required",
None => "in plaintext",
},
if cluster.master_tls.is_some() { "over TLS" } else { "in plaintext" },
);
// Registered only now, with storage connected and the listener bound,
// so the gateway never finds an engine that cannot serve yet.
let advertise = advertised_addr(&args.listen, &cluster.masters, args.advertise.as_deref());
tokio::spawn(master_client::keep_connected(
cluster.masters.clone(),
advertise,
args.filer_group.clone(),
args.metrics_port,
cluster.master_tls.clone(),
metrics.clone(),
));
let router = Server::builder()
.add_service(health_service)
// Match the gateway's 1 GiB cap — put batches over the tonic 4 MiB
// default otherwise get rejected before they reach the handler.
.add_service(
VectorServiceServer::new(service)
.max_decoding_message_size(1 << 30)
.max_encoding_message_size(1 << 30),
)
.serve_with_incoming(incoming)
.await?;
VectorServiceServer::new(ObservedService {
inner: service,
metrics,
})
.max_decoding_message_size(1 << 30)
.max_encoding_message_size(1 << 30),
);
match server_tls {
Some(tls) => {
tokio::spawn(tls.clone().refresh(tls::refresh_interval()));
router.serve_with_incoming(tls.incoming(listener)).await?;
}
None => {
router
.serve_with_incoming(TcpListenerStream::new(listener))
.await?
}
}
Ok(())
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn an_empty_advertise_uses_the_listen_address() {
for value in [None, Some(""), Some(" ")] {
assert_eq!(
advertised_addr("127.0.0.1:8090", &[], value),
"127.0.0.1:8090"
);
}
assert_eq!(
advertised_addr("127.0.0.1:8090", &[], Some("vector.example:8090")),
"vector.example:8090"
);
}
#[test]
fn nprobes_must_be_positive() {
let parse = |nprobes: &str| {
Args::try_parse_from(["weed-vector", "--masters", "m:9333", "--nprobes", nprobes])
};
assert!(parse("0").is_err());
let args = parse("12").unwrap();
assert_eq!(args.nprobes.map(std::num::NonZeroUsize::get), Some(12));
}
#[test]
fn the_catalog_flags_are_gone() {
let parse = |extra: &[&str]| {
let mut argv = vec!["weed-vector", "-master", "m:9333"];
argv.extend_from_slice(extra);
Args::try_parse_from(normalize_args(argv.into_iter().map(String::from).collect()))
};
assert_eq!(parse(&[]).unwrap().masters, "m:9333");
// The gateway resolves every table it sends; the engine has no
// catalog to name or authenticate to.
assert!(parse(&["--namespace", "http://localhost:9101"]).is_err());
assert!(parse(&["-filer-signing-key", "k"]).is_err());
}
#[test]
fn the_masters_are_required() {
assert!(Args::try_parse_from(["weed-vector"]).is_err());
}
}
+162 -53
View File
@@ -7,53 +7,26 @@ use std::time::Duration;
use anyhow::{Context, Result};
use tokio::time::sleep;
use tonic::transport::Endpoint;
use crate::master_pb::{KeepConnectedRequest, seaweed_client::SeaweedClient};
use crate::master_pb::{
KeepConnectedRequest, ListClusterNodesRequest, seaweed_client::SeaweedClient,
};
use crate::security::GrpcTls;
pub use weed_lance_worker::cluster::{CONNECT_TIMEOUT, connect, grpc_addr};
fn registration(client_addr: &str) -> KeepConnectedRequest {
fn registration(client_addr: &str, filer_group: &str, metrics_port: u16) -> KeepConnectedRequest {
KeepConnectedRequest {
client_type: "vector".into(),
client_address: client_addr.into(),
version: env!("CARGO_PKG_VERSION").into(),
filer_group: String::new(),
filer_group: filer_group.into(),
data_center: String::new(),
rack: String::new(),
leave_lock_ring: false,
..Default::default()
metrics_port: metrics_port.into(),
}
}
/// Masters are configured by their HTTP address; gRPC lives +10000 like the
/// weed conventions (server_address.ToGrpcAddress).
// A master address may carry an explicit gRPC port as `host:http.grpc` —
// use it directly; otherwise the gRPC port is http + 10000.
pub fn grpc_addr(master: &str) -> Result<String> {
let (scheme, rest) = master.split_once("://").unwrap_or(("http", master));
if let Some((host, port)) = rest.rsplit_once(':') {
if let Some((_, grpc)) = port.split_once('.') {
if let Ok(grpc) = grpc.parse::<u64>() {
let grpc = u16::try_from(grpc)
.with_context(|| format!("master {master}: gRPC port {grpc} is past 65535"))?;
return Ok(format!("{scheme}://{host}:{grpc}"));
}
}
if let Ok(port) = port.parse::<u64>() {
let grpc = port
.checked_add(10000)
.and_then(|grpc| u16::try_from(grpc).ok())
.with_context(|| {
format!(
"master {master}: gRPC port {port}+10000 is past 65535; \
give it explicitly as {host}:{port}.<grpc port>"
)
})?;
return Ok(format!("{scheme}://{host}:{grpc}"));
}
}
Ok(format!("{scheme}://{rest}"))
}
// `host:http.grpc` normalizes to `host:http` so a leader hint naming the
// dialed master does not read as a redirect.
fn http_addr(master: &str) -> String {
@@ -68,11 +41,22 @@ fn http_addr(master: &str) -> String {
/// Retry forever: try each master until a stream opens, hold it until it
/// errors, then rediscover — same shape as wdclient.KeepConnectedToMaster.
pub async fn keep_connected(masters: Vec<String>, client_addr: String) {
let client_addr = client_addr.as_str();
pub async fn keep_connected(
masters: Vec<String>,
client_addr: String,
filer_group: String,
metrics_port: u16,
tls: Option<GrpcTls>,
metrics: crate::metrics::Metrics,
) {
let registration = &registration(&client_addr, &filer_group, metrics_port);
let tls = tls.as_ref();
let metrics = &metrics;
loop {
try_all_masters(&masters, |master| async move {
stream_to(&master, client_addr).await
let result = stream_to(&master, registration, tls, Some(&metrics)).await;
metrics.set_connected(false).await;
result
})
.await;
sleep(Duration::from_secs(3)).await;
@@ -108,21 +92,18 @@ where
}
}
/// Bounds the TCP connect and then the master's first answer, the 5s wdclient
/// gives a master call; a black-holed or silent master otherwise holds up the
/// other masters.
const CONNECT_TIMEOUT: Duration = Duration::from_secs(5);
async fn stream_to(master: &str, client_addr: &str) -> Result<Option<String>> {
let channel = Endpoint::new(grpc_addr(master)?)?
.connect_timeout(CONNECT_TIMEOUT)
.connect()
.await?;
async fn stream_to(
master: &str,
registration: &KeepConnectedRequest,
tls: Option<&GrpcTls>,
metrics: Option<&crate::metrics::Metrics>,
) -> Result<Option<String>> {
let channel = connect(master, tls).await?;
// A master answers at once, a follower with its leader hint.
let answer_by = tokio::time::Instant::now() + CONNECT_TIMEOUT;
let mut client = SeaweedClient::new(channel);
let (tx, rx) = tokio::sync::mpsc::channel(8);
tx.send(registration(client_addr)).await?;
tx.send(registration.clone()).await?;
let mut stream = tokio::time::timeout_at(
answer_by,
client.keep_connected(tokio_stream::wrappers::ReceiverStream::new(rx)),
@@ -131,6 +112,8 @@ async fn stream_to(master: &str, client_addr: &str) -> Result<Option<String>> {
.context("master did not answer keep-connected")??
.into_inner();
let mut answered = false;
let interval = Duration::from_secs(5);
let mut heartbeat = tokio::time::interval_at(tokio::time::Instant::now() + interval, interval);
// The master keeps the node live while the stream is open; resend the
// registration periodically so a reconnect never leaves a stale entry.
loop {
@@ -149,23 +132,72 @@ async fn stream_to(master: &str, client_addr: &str) -> Result<Option<String>> {
return Ok(Some(loc.leader));
}
}
if let Some(metrics) = metrics { metrics.set_connected(true).await; }
}
None => anyhow::bail!("master closed the keep-connected stream"),
},
_ = tokio::time::sleep_until(answer_by), if !answered => {
anyhow::bail!("master did not answer keep-connected");
}
_ = sleep(Duration::from_secs(5)) => {
tx.send(registration(client_addr)).await?;
_ = heartbeat.tick() => {
tokio::time::timeout(CONNECT_TIMEOUT, tx.send(registration.clone())).await
.context("registration send stalled")??;
probe_registration(&mut client).await?;
}
}
}
}
async fn probe_registration(client: &mut SeaweedClient<tonic::transport::Channel>) -> Result<()> {
tokio::time::timeout(
CONNECT_TIMEOUT,
client.list_cluster_nodes(ListClusterNodesRequest {
client_type: "vector".into(),
..Default::default()
}),
)
.await
.context("registration heartbeat stalled")??;
Ok(())
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn registration_preserves_filer_group() {
assert_eq!(
registration("vector:8090", "tenant-a", 0).filer_group,
"tenant-a"
);
}
#[test]
fn registration_advertises_metrics_port() {
assert_eq!(registration("vector:8090", "", 9327).metrics_port, 9327);
assert_eq!(registration("vector:8090", "", 0).metrics_port, 0);
}
#[tokio::test]
async fn stalled_registration_heartbeat_is_bounded() {
let listener = tokio::net::TcpListener::bind("127.0.0.1:0").await.unwrap();
let address = listener.local_addr().unwrap();
let hold = tokio::spawn(async move {
let (_connection, _) = listener.accept().await.unwrap();
std::future::pending::<()>().await;
});
let channel = tonic::transport::Endpoint::from_shared(format!("http://{address}"))
.unwrap()
.connect_lazy();
let mut client = SeaweedClient::new(channel);
let result = tokio::time::timeout(CONNECT_TIMEOUT * 3, probe_registration(&mut client))
.await
.unwrap();
assert!(format!("{:#}", result.unwrap_err()).contains("registration heartbeat stalled"));
hold.abort();
}
/// Runs one pass against scripted masters and returns the dial order.
async fn pass(masters: &[&str], answer: fn(&str) -> Result<Option<String>>) -> Vec<String> {
let masters: Vec<String> = masters.iter().map(|m| m.to_string()).collect();
@@ -215,7 +247,8 @@ mod tests {
#[tokio::test]
async fn dial_to_a_black_holed_master_is_bounded() {
// TEST-NET-1 (RFC 5737) is reserved, so nothing answers the SYN.
let dial = stream_to("192.0.2.1:9333", "127.0.0.1:8080");
let registration = registration("127.0.0.1:8080", "", 0);
let dial = stream_to("192.0.2.1:9333", &registration, None, None);
let result = tokio::time::timeout(CONNECT_TIMEOUT * 3, dial).await;
assert!(matches!(result, Ok(Err(_))), "dial was not bounded");
}
@@ -229,13 +262,89 @@ mod tests {
let _ = tokio::io::copy(&mut conn, &mut tokio::io::sink()).await;
});
let master = format!("127.0.0.1:9333.{port}");
let open = stream_to(&master, "127.0.0.1:8080");
let registration = registration("127.0.0.1:8080", "", 0);
let open = stream_to(&master, &registration, None, None);
match tokio::time::timeout(CONNECT_TIMEOUT * 3, open).await {
Ok(Err(e)) => assert!(format!("{e:#}").contains("did not answer"), "{e:#}"),
other => panic!("open was not bounded: {other:?}"),
}
}
/// The worker crate's `tests/tls`: a CA and the `engine-a` leaf it signed.
fn credentials() -> GrpcTls {
let fixture = |name: &str| {
std::path::Path::new(env!("CARGO_MANIFEST_DIR"))
.join("../../../seaweed-worker/crates/lance/tests/tls")
.join(name)
.display()
.to_string()
};
GrpcTls {
ca_path: fixture("ca.pem"),
cert_path: fixture("a.pem"),
key_path: fixture("a.key"),
allowed_server_names: Vec::new(),
allowed_server_wildcard_domain: String::new(),
}
}
#[tokio::test]
async fn a_master_that_stalls_the_tls_handshake_is_bounded() {
let listener = tokio::net::TcpListener::bind("127.0.0.1:0").await.unwrap();
let port = listener.local_addr().unwrap().port();
tokio::spawn(async move {
let (mut conn, _) = listener.accept().await.unwrap();
let _ = tokio::io::copy(&mut conn, &mut tokio::io::sink()).await;
});
let master = format!("127.0.0.1:9333.{port}");
let tls = credentials();
let registration = registration("127.0.0.1:8080", "", 0);
let open = stream_to(&master, &registration, Some(&tls), None);
match tokio::time::timeout(CONNECT_TIMEOUT * 3, open).await {
Ok(Err(e)) => assert!(format!("{e:#}").contains("did not complete"), "{e:#}"),
other => panic!("handshake was not bounded: {other:?}"),
}
}
#[cfg(unix)]
#[tokio::test]
async fn a_credential_read_that_stalls_is_bounded() {
let dir = tempfile::tempdir().unwrap();
let fifo = dir.path().join("ca.pem");
let made = std::process::Command::new("mkfifo").arg(&fifo).status();
assert!(made.unwrap().success());
// Every connection is dropped at once, so a dial that skipped the read
// fails its handshake instead of timing out.
let listener = tokio::net::TcpListener::bind("127.0.0.1:0").await.unwrap();
let port = listener.local_addr().unwrap().port();
tokio::spawn(async move {
while let Ok((conn, _)) = listener.accept().await {
drop(conn);
}
});
let mut tls = credentials();
tls.ca_path = fifo.display().to_string();
let master = format!("127.0.0.1:9333.{port}");
let registration = registration("127.0.0.1:8080", "", 0);
let open = stream_to(&master, &registration, Some(&tls), None);
let result = tokio::time::timeout(CONNECT_TIMEOUT * 3, open).await;
use std::io::Write;
let mut cleanup = std::fs::OpenOptions::new()
.read(true)
.write(true)
.open(&fifo)
.unwrap();
let replacement = dir.path().join("ca.replacement");
std::fs::write(&replacement, b"invalid CA").unwrap();
std::fs::rename(&replacement, &fifo).unwrap();
cleanup.write_all(b"invalid CA").unwrap();
drop(cleanup);
match result {
Ok(Err(e)) => assert!(format!("{e:#}").contains("did not complete"), "{e:#}"),
other => panic!("credential read was not bounded: {other:?}"),
}
}
#[test]
fn grpc_addr_refuses_a_port_past_65535() {
assert_eq!(grpc_addr("m1:9333").unwrap(), "http://m1:19333");
+341
View File
@@ -0,0 +1,341 @@
use std::net::SocketAddr;
use anyhow::Result;
use axum::{Router, http::StatusCode, routing::get};
use prometheus::{
Encoder, Histogram, HistogramOpts, HistogramVec, IntCounter, IntCounterVec, IntGauge,
IntGaugeVec, Opts, Registry, TextEncoder,
};
use tonic_health::{ServingStatus, server::HealthReporter};
use crate::read_cache::ReadCache;
#[derive(Clone)]
pub struct Metrics {
pub registry: Registry,
pub connected: IntGauge,
pub requests: IntCounterVec,
pub request_seconds: HistogramVec,
pub in_flight: IntGaugeVec,
pub query_results: IntCounter,
pub cache_requests: IntCounterVec,
pub storage_bytes: IntCounter,
pub storage_read_seconds: Histogram,
cache_bytes: IntGauge,
cache_entries: IntGauge,
health: Option<HealthReporter>,
}
impl Metrics {
pub fn new() -> Result<Self> {
let registry = Registry::new();
let opts = |name: &str, help: &str| {
Opts::new(name, help)
.namespace("SeaweedFS")
.subsystem("vector")
};
let buckets = prometheus::exponential_buckets(0.0001, 2.0, 24)?;
let histogram =
|name: &str, help: &str| HistogramOpts::from(opts(name, help)).buckets(buckets.clone());
let connected = IntGauge::with_opts(opts(
"connected",
"Whether the engine is registered with the master.",
))?;
let requests = IntCounterVec::new(
opts(
"request_total",
"Vector engine requests by operation and gRPC status.",
),
&["type", "code"],
)?;
let request_seconds = HistogramVec::new(
histogram("request_seconds", "Vector engine request duration."),
&["type"],
)?;
let in_flight = IntGaugeVec::new(
opts("in_flight_requests", "Vector engine requests in progress."),
&["type"],
)?;
let query_results =
IntCounter::with_opts(opts("query_results_total", "Vectors returned by queries."))?;
let cache_requests = IntCounterVec::new(
opts("cache_requests_total", "Immutable reads by cache outcome."),
&["result"],
)?;
let storage_bytes = IntCounter::with_opts(opts(
"storage_read_bytes_total",
"Bytes fetched for immutable cache reads.",
))?;
let storage_read_seconds = Histogram::with_opts(histogram(
"storage_read_seconds",
"Immutable storage fetch duration.",
))?;
let cache_bytes =
IntGauge::with_opts(opts("cache_bytes", "Current cache weight in bytes."))?;
let cache_entries = IntGauge::with_opts(opts("cache_entries", "Current cache entries."))?;
for collector in [
Box::new(connected.clone()) as Box<dyn prometheus::core::Collector>,
Box::new(requests.clone()),
Box::new(request_seconds.clone()),
Box::new(in_flight.clone()),
Box::new(query_results.clone()),
Box::new(cache_requests.clone()),
Box::new(storage_bytes.clone()),
Box::new(storage_read_seconds.clone()),
Box::new(cache_bytes.clone()),
Box::new(cache_entries.clone()),
] {
registry.register(collector)?;
}
Ok(Self {
registry,
connected,
requests,
request_seconds,
in_flight,
query_results,
cache_requests,
storage_bytes,
storage_read_seconds,
cache_bytes,
cache_entries,
health: None,
})
}
pub fn with_health(mut self, health: HealthReporter) -> Self {
self.health = Some(health);
self
}
pub async fn set_connected(&self, connected: bool) {
self.connected.set(i64::from(connected));
if let Some(health) = self.health.clone() {
let status = if connected {
ServingStatus::Serving
} else {
ServingStatus::NotServing
};
health.set_service_status("", status).await;
health
.set_service_status("vector_pb.VectorService", status)
.await;
}
}
pub fn start(&self, operation: &str) -> InFlight {
let gauge = self.in_flight.with_label_values(&[operation]);
gauge.inc();
InFlight {
gauge,
requests: self.requests.clone(),
operation: operation.into(),
completed: false,
_timer: self
.request_seconds
.with_label_values(&[operation])
.start_timer(),
}
}
fn gather(&self, cache: Option<&ReadCache>) -> Result<String> {
if let Some(cache) = cache {
let (entries, bytes) = cache.size();
self.cache_entries.set(entries as i64);
self.cache_bytes.set(bytes as i64);
}
let mut buffer = Vec::new();
TextEncoder::new().encode(&self.registry.gather(), &mut buffer)?;
Ok(String::from_utf8(buffer)?)
}
}
pub struct InFlight {
gauge: IntGauge,
requests: IntCounterVec,
operation: String,
completed: bool,
_timer: prometheus::HistogramTimer,
}
impl InFlight {
pub fn complete(&mut self, error: Option<tonic::Code>) {
let code = error
.map(|c| format!("{c:?}"))
.unwrap_or_else(|| "OK".into());
self.requests
.with_label_values(&[&self.operation, &code])
.inc();
self.completed = true;
}
}
impl Drop for InFlight {
fn drop(&mut self) {
self.gauge.dec();
if !self.completed {
self.requests
.with_label_values(&[&self.operation, "Cancelled"])
.inc();
}
}
}
pub async fn serve(metrics: Metrics, address: SocketAddr, cache: Option<ReadCache>) -> Result<()> {
let listener = tokio::net::TcpListener::bind(address).await?;
serve_listener(metrics, listener, cache).await
}
pub async fn serve_listener(
metrics: Metrics,
listener: tokio::net::TcpListener,
cache: Option<ReadCache>,
) -> Result<()> {
let readiness = metrics.clone();
let app = Router::new()
.route("/health", get(|| async { StatusCode::OK }))
.route(
"/ready",
get(move || {
let metrics = readiness.clone();
async move {
if metrics.connected.get() == 1 {
StatusCode::OK
} else {
StatusCode::SERVICE_UNAVAILABLE
}
}
}),
)
.route(
"/metrics",
get(move || {
let metrics = metrics.clone();
let cache = cache.clone();
async move {
metrics
.gather(cache.as_ref())
.map(|body| ([("content-type", "text/plain; version=0.0.4")], body))
.map_err(|_| StatusCode::INTERNAL_SERVER_ERROR)
}
}),
);
axum::serve(listener, app).await?;
Ok(())
}
#[cfg(test)]
mod tests {
use super::*;
#[tokio::test]
async fn endpoints_follow_registration_and_publish_metrics() {
let metrics = Metrics::new().unwrap();
let listener = tokio::net::TcpListener::bind("127.0.0.1:0").await.unwrap();
let endpoint = format!("http://{}", listener.local_addr().unwrap());
let task = tokio::spawn(serve_listener(metrics.clone(), listener, None));
let client = reqwest::Client::new();
assert_eq!(
client
.get(format!("{endpoint}/health"))
.send()
.await
.unwrap()
.status(),
200
);
assert_eq!(
client
.get(format!("{endpoint}/ready"))
.send()
.await
.unwrap()
.status(),
503
);
metrics.set_connected(true).await;
assert_eq!(
client
.get(format!("{endpoint}/ready"))
.send()
.await
.unwrap()
.status(),
200
);
let in_flight = metrics.start("QueryVectors");
let body = client
.get(format!("{endpoint}/metrics"))
.send()
.await
.unwrap()
.text()
.await
.unwrap();
assert!(body.contains("SeaweedFS_vector_connected 1"));
assert!(body.contains("SeaweedFS_vector_in_flight_requests{type=\"QueryVectors\"} 1"));
drop(in_flight);
assert_eq!(
metrics.in_flight.with_label_values(&["QueryVectors"]).get(),
0
);
metrics.set_connected(false).await;
assert_eq!(
client
.get(format!("{endpoint}/ready"))
.send()
.await
.unwrap()
.status(),
503
);
task.abort();
}
#[tokio::test]
async fn grpc_health_follows_registration() {
use tonic_health::pb::{HealthCheckRequest, health_client::HealthClient};
let (reporter, service) = tonic_health::server::health_reporter();
let metrics = Metrics::new().unwrap().with_health(reporter);
metrics.set_connected(false).await;
let listener = tokio::net::TcpListener::bind("127.0.0.1:0").await.unwrap();
let endpoint = format!("http://{}", listener.local_addr().unwrap());
let task = tokio::spawn(
tonic::transport::Server::builder()
.add_service(service)
.serve_with_incoming(tokio_stream::wrappers::TcpListenerStream::new(listener)),
);
let channel = tonic::transport::Endpoint::from_shared(endpoint)
.unwrap()
.connect()
.await
.unwrap();
let mut client = HealthClient::new(channel);
let request = || HealthCheckRequest {
service: "vector_pb.VectorService".into(),
};
assert_eq!(
client.check(request()).await.unwrap().into_inner().status,
2
);
metrics.set_connected(true).await;
assert_eq!(
client.check(request()).await.unwrap().into_inner().status,
1
);
metrics.set_connected(false).await;
assert_eq!(
client.check(request()).await.unwrap().into_inner().status,
2
);
task.abort();
}
}
impl std::fmt::Debug for Metrics {
fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
f.debug_struct("Metrics")
.field("connected", &self.connected.get())
.finish()
}
}
@@ -0,0 +1,122 @@
use crate::metrics::Metrics;
use crate::pb::vector::{self as pb, vector_service_server::VectorService};
use crate::service::VectorServer;
use tonic::{Request, Response, Status};
pub struct ObservedService {
pub inner: VectorServer,
pub metrics: Metrics,
}
#[tonic::async_trait]
impl VectorService for ObservedService {
async fn create_index(
&self,
request: Request<pb::CreateIndexRequest>,
) -> Result<Response<pb::CreateIndexResponse>, Status> {
let mut in_flight = self.metrics.start("CreateIndex");
let result = self.inner.create_index(request).await;
in_flight.complete(result.as_ref().err().map(|e| e.code()));
result
}
async fn drop_index(
&self,
request: Request<pb::DropIndexRequest>,
) -> Result<Response<pb::DropIndexResponse>, Status> {
let mut in_flight = self.metrics.start("DropIndex");
let result = self.inner.drop_index(request).await;
in_flight.complete(result.as_ref().err().map(|e| e.code()));
result
}
async fn put_vectors(
&self,
request: Request<pb::PutVectorsRequest>,
) -> Result<Response<pb::PutVectorsResponse>, Status> {
let mut in_flight = self.metrics.start("PutVectors");
let result = self.inner.put_vectors(request).await;
in_flight.complete(result.as_ref().err().map(|e| e.code()));
result
}
async fn get_vectors(
&self,
request: Request<pb::GetVectorsRequest>,
) -> Result<Response<pb::GetVectorsResponse>, Status> {
let mut in_flight = self.metrics.start("GetVectors");
let result = self.inner.get_vectors(request).await;
in_flight.complete(result.as_ref().err().map(|e| e.code()));
result
}
async fn delete_vectors(
&self,
request: Request<pb::DeleteVectorsRequest>,
) -> Result<Response<pb::DeleteVectorsResponse>, Status> {
let mut in_flight = self.metrics.start("DeleteVectors");
let result = self.inner.delete_vectors(request).await;
in_flight.complete(result.as_ref().err().map(|e| e.code()));
result
}
async fn list_vectors(
&self,
request: Request<pb::ListVectorsRequest>,
) -> Result<Response<pb::ListVectorsResponse>, Status> {
let mut in_flight = self.metrics.start("ListVectors");
let result = self.inner.list_vectors(request).await;
in_flight.complete(result.as_ref().err().map(|e| e.code()));
result
}
async fn query_vectors(
&self,
request: Request<pb::QueryVectorsRequest>,
) -> Result<Response<pb::QueryVectorsResponse>, Status> {
let mut in_flight = self.metrics.start("QueryVectors");
let result = self.inner.query_vectors(request).await;
in_flight.complete(result.as_ref().err().map(|e| e.code()));
if let Ok(response) = &result {
self.metrics
.query_results
.inc_by(response.get_ref().vectors.len() as u64);
}
result
}
}
#[cfg(test)]
mod tests {
use super::*;
#[tokio::test]
async fn failures_are_counted_and_release_in_flight_requests() {
let service = ObservedService {
inner: VectorServer::new(),
metrics: Metrics::new().unwrap(),
};
let response = service
.query_vectors(Request::new(pb::QueryVectorsRequest::default()))
.await;
let code = format!("{:?}", response.unwrap_err().code());
assert_eq!(
service
.metrics
.requests
.with_label_values(&["QueryVectors", &code])
.get(),
1
);
assert_eq!(
service
.metrics
.in_flight
.with_label_values(&["QueryVectors"])
.get(),
0
);
assert_eq!(
service
.metrics
.request_seconds
.with_label_values(&["QueryVectors"])
.get_sample_count(),
1
);
}
}
+223 -49
View File
@@ -9,6 +9,7 @@ use std::fmt;
use std::ops::Range;
use std::sync::Arc;
use crate::metrics::Metrics;
use bytes::Bytes;
use futures::StreamExt;
use futures::stream::BoxStream;
@@ -37,26 +38,53 @@ struct Entry {
/// Wraps each object store Lance opens; clones share one cache.
#[derive(Clone)]
pub struct ReadCache(moka::sync::Cache<Key, Entry>);
pub struct ReadCache {
cache: moka::sync::Cache<Key, Entry>,
/// Keeps entries apart from those of another [`Self::scoped`] view, for
/// stores that share a prefix but not their bytes.
scope: Arc<str>,
metrics: Option<Metrics>,
}
impl ReadCache {
pub fn new(capacity_bytes: u64) -> Self {
Self(
moka::sync::Cache::builder()
Self {
cache: moka::sync::Cache::builder()
.max_capacity(capacity_bytes)
.weigher(|key: &Key, entry: &Entry| {
u32::try_from(key.path.as_ref().len() + entry.bytes.len()).unwrap_or(u32::MAX)
})
.build(),
)
scope: "".into(),
metrics: None,
}
}
pub fn with_metrics(mut self, metrics: Metrics) -> Self {
self.metrics = Some(metrics);
self
}
pub fn size(&self) -> (u64, u64) {
self.cache.run_pending_tasks();
(self.cache.entry_count(), self.cache.weighted_size())
}
/// The same cache, keyed apart under `scope`.
pub fn scoped(&self, scope: &str) -> Self {
Self {
cache: self.cache.clone(),
scope: scope.into(),
metrics: self.metrics.clone(),
}
}
}
impl fmt::Debug for ReadCache {
fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
f.debug_struct("ReadCache")
.field("entries", &self.0.entry_count())
.field("bytes", &self.0.weighted_size())
.field("entries", &self.cache.entry_count())
.field("bytes", &self.cache.weighted_size())
.finish()
}
}
@@ -65,8 +93,9 @@ impl WrappingObjectStore for ReadCache {
fn wrap(&self, store_prefix: &str, original: Arc<dyn ObjectStore>) -> Arc<dyn ObjectStore> {
Arc::new(CachedStore {
inner: original,
store: store_prefix.into(),
cache: self.0.clone(),
store: format!("{}|{store_prefix}", self.scope).into(),
cache: self.cache.clone(),
metrics: self.metrics.clone(),
})
}
}
@@ -122,6 +151,7 @@ struct CachedStore {
inner: Arc<dyn ObjectStore>,
store: Arc<str>,
cache: moka::sync::Cache<Key, Entry>,
metrics: Option<Metrics>,
}
impl CachedStore {
@@ -170,14 +200,47 @@ impl ObjectStore for CachedStore {
object: Some(object),
}) = self.cache.get(&key)
{
if let Some(metrics) = &self.metrics {
metrics.cache_requests.with_label_values(&["hit"]).inc();
}
return Ok(whole(bytes, &object));
}
if let Some(metrics) = &self.metrics {
metrics.cache_requests.with_label_values(&["miss"]).inc();
}
let _timer = self
.metrics
.as_ref()
.map(|m| m.storage_read_seconds.start_timer());
let result = self.inner.get_opts(location, options).await?;
if result.meta.size > MAX_WHOLE_OBJECT {
return Ok(result);
let meta = result.meta.clone();
let range = result.range.clone();
let attributes = result.attributes.clone();
let metrics = self.metrics.clone();
let timer = _timer;
let stream = result
.into_stream()
.map(move |chunk| {
let _timer = &timer;
if let (Some(metrics), Ok(bytes)) = (&metrics, &chunk) {
metrics.storage_bytes.inc_by(bytes.len() as u64);
}
chunk
})
.boxed();
return Ok(GetResult {
payload: GetResultPayload::Stream(stream),
meta,
range,
attributes,
});
}
let object = Arc::new((result.meta.clone(), result.attributes.clone()));
let bytes = owned(result.bytes().await?);
if let Some(metrics) = &self.metrics {
metrics.storage_bytes.inc_by(bytes.len() as u64);
}
self.cache.insert(
key,
Entry {
@@ -206,7 +269,21 @@ impl ObjectStore for CachedStore {
.filter(|(_, hit)| hit.is_none())
.map(|(range, _)| range.clone())
.collect();
if let Some(metrics) = &self.metrics {
metrics
.cache_requests
.with_label_values(&["miss"])
.inc_by(missing.len() as u64);
metrics
.cache_requests
.with_label_values(&["hit"])
.inc_by((ranges.len() - missing.len()) as u64);
}
if !missing.is_empty() {
let _timer = self
.metrics
.as_ref()
.map(|m| m.storage_read_seconds.start_timer());
let mut fetched = self.inner.get_ranges(location, &missing).await?.into_iter();
for (range, slot) in ranges.iter().zip(out.iter_mut()) {
if slot.is_some() {
@@ -216,6 +293,9 @@ impl ObjectStore for CachedStore {
break;
};
let bytes = owned(bytes);
if let Some(metrics) = &self.metrics {
metrics.storage_bytes.inc_by(bytes.len() as u64);
}
self.cache.insert(
self.key(location, Some(range.clone())),
Entry {
@@ -273,12 +353,10 @@ mod tests {
use super::*;
use crate::pb::vector as pb;
use crate::pb::vector::vector_service_server::VectorService;
use crate::service::VectorServer;
use crate::service::{VectorServer, schema_for, table_id};
use object_store::ObjectStoreExt;
use object_store::memory::InMemory;
use tonic::Request;
use weed_lance_worker::catalog::NamespaceClient;
use weed_lance_worker::dataset::FallbackOptions;
fn cached(inner: Arc<InMemory>) -> Arc<dyn ObjectStore> {
ReadCache::new(1 << 20).wrap("memory", inner)
@@ -373,8 +451,8 @@ mod tests {
.unwrap();
read(&store, &path, 0..(8 << 10)).await;
}
cache.0.run_pending_tasks();
assert!(cache.0.weighted_size() <= 64 << 10, "{cache:?}");
cache.cache.run_pending_tasks();
assert!(cache.cache.weighted_size() <= 64 << 10, "{cache:?}");
}
#[tokio::test]
@@ -392,40 +470,15 @@ mod tests {
let backing = whole.as_ptr_range();
read(&store, &path, 10..20).await;
let key = Key {
store: "memory".into(),
store: "|memory".into(),
path,
range: Some(10..20),
};
let cached = cache.0.get(&key).unwrap().bytes;
let cached = cache.cache.get(&key).unwrap().bytes;
assert_eq!(cached.len(), 10);
assert!(!backing.contains(&cached.as_ptr()));
}
/// A catalog that answers every request with `location`.
async fn catalog_at(location: String) -> String {
use tokio::io::{AsyncReadExt, AsyncWriteExt};
let listener = tokio::net::TcpListener::bind("127.0.0.1:0").await.unwrap();
let url = format!("http://{}", listener.local_addr().unwrap());
let body = serde_json::json!({ "location": location }).to_string();
tokio::spawn(async move {
loop {
let (mut conn, _) = listener.accept().await.unwrap();
let body = body.clone();
tokio::spawn(async move {
let mut request = [0u8; 8192];
let _ = conn.read(&mut request).await;
let response = format!(
"HTTP/1.1 200 OK\r\ncontent-type: application/json\r\ncontent-length: {}\r\nconnection: close\r\n\r\n{body}",
body.len()
);
let _ = conn.write_all(response.as_bytes()).await;
let _ = conn.shutdown().await;
});
}
});
url
}
fn vector(key: &str, data: Vec<f32>) -> pb::Vector {
pb::Vector {
key: key.to_string(),
@@ -492,18 +545,28 @@ mod tests {
/// dropped and recreated at the same location answers with its new rows.
#[tokio::test]
async fn cached_reads_follow_writes_and_a_recreated_index() {
let location = format!("shared-memory://cache-{}/vector/x", uuid_like(),);
let server = VectorServer::new(
NamespaceClient::new(catalog_at(location.clone()).await),
FallbackOptions::new(),
)
.with_read_cache(ReadCache::new(16 << 20));
let bucket = format!("cache-{}", uuid_like());
let location = format!("shared-memory://{bucket}/vector/x");
let server = VectorServer::new()
.with_schemes(&["shared-memory"])
.with_read_cache(ReadCache::new(16 << 20));
let table = pb::Table {
id: vec!["bkt".into(), "vector".into(), "x".into()],
version: 0,
id: vec![bucket, "vector".into(), "x".into()],
location: location.clone(),
..Default::default()
};
create(&server, &table, 4).await;
create(&server, &table, 4).await;
let mismatch = server
.create_index(Request::new(pb::CreateIndexRequest {
table: Some(table.clone()),
dimension: 3,
..Default::default()
}))
.await
.unwrap_err();
assert_eq!(mismatch.code(), tonic::Code::AlreadyExists);
put(
&server,
&table,
@@ -553,6 +616,117 @@ mod tests {
assert_eq!(query(&server, &table, vec![1.0, 0.0]).await, ["z"]);
}
/// A query loads the index once; the next one finds it in the session
/// every read shares.
#[tokio::test]
async fn queries_reuse_the_loaded_index() {
let bucket = format!("session-{}", uuid_like());
let location = format!("shared-memory://{bucket}/vector/x");
let server = VectorServer::new().with_schemes(&["shared-memory"]);
let table = pb::Table {
id: vec![bucket, "vector".into(), "x".into()],
location: location.clone(),
..Default::default()
};
create(&server, &table, 2).await;
put(
&server,
&table,
(0..300)
.map(|i| vector(&format!("k{i}"), vec![i as f32, 1.0]))
.collect(),
)
.await;
let mut dataset = lance::Dataset::open(&location).await.unwrap();
weed_lance_worker::vector_index::build(&mut dataset, "vector", "idx", 2, false)
.await
.unwrap();
assert_eq!(query(&server, &table, vec![5.0, 1.0]).await[0], "k5");
let session = server.reads.lock().unwrap().session.clone();
let loaded = session.index_cache_stats().await;
assert!(session.approx_num_items() > 0);
assert_eq!(query(&server, &table, vec![7.0, 1.0]).await[0], "k7");
let again = session.index_cache_stats().await;
assert!(server.handles.contains_key(&(location, false)));
assert!(again.hits > loaded.hits, "{loaded:?} then {again:?}");
assert_eq!(again.misses, loaded.misses, "{loaded:?} then {again:?}");
}
/// An index dropped and created again at the same location reaches the
/// version numbers of the old one; the shared session must not answer
/// for them with the old index.
#[tokio::test]
async fn a_recreated_table_does_not_see_the_old_index() {
recreate_after(false).await;
recreate_after(true).await;
}
/// `unmarked`: the first table is written without a table id, as indexes
/// created before ids were are.
async fn recreate_after(unmarked: bool) {
let bucket = format!("recreate-{}", uuid_like());
let location = format!("shared-memory://{bucket}/vector/x");
let server = VectorServer::new().with_schemes(&["shared-memory"]);
let table = pb::Table {
id: vec![bucket, "vector".into(), "x".into()],
location: location.clone(),
..Default::default()
};
let rows = |offset: usize| {
(0..300)
.map(|i| vector(&format!("k{}", i + offset), vec![(i + offset) as f32, 1.0]))
.collect::<Vec<_>>()
};
if unmarked {
let schema = schema_for(2, pb::DistanceMetric::L2, &[]);
let empty = arrow_array::RecordBatch::new_empty(schema.clone());
let reader = arrow_array::RecordBatchIterator::new(vec![Ok(empty)], schema);
lance::Dataset::write(reader, &location, None)
.await
.unwrap();
} else {
create(&server, &table, 2).await;
}
put(&server, &table, rows(0)).await;
let mut dataset = lance::Dataset::open(&location).await.unwrap();
assert_eq!(table_id(&dataset).is_empty(), unmarked);
weed_lance_worker::vector_index::build(&mut dataset, "vector", "idx", 2, false)
.await
.unwrap();
let indexed = dataset.version().version;
assert_eq!(query(&server, &table, vec![5.0, 1.0]).await[0], "k5");
let other = pb::Table {
id: vec![table.id[0].clone(), "vector".into(), "y".into()],
location: location.replace("/vector/x", "/vector/y"),
..Default::default()
};
create(&server, &other, 2).await;
put(&server, &other, rows(0)).await;
assert_eq!(query(&server, &other, vec![5.0, 1.0]).await[0], "k5");
let (store, path) = lance::io::ObjectStore::from_uri(&location).await.unwrap();
store.remove_dir_all(path).await.unwrap();
create(&server, &table, 2).await;
let mut offset = 1000;
while lance::Dataset::open(&location)
.await
.unwrap()
.version()
.version
< indexed
{
put(&server, &table, rows(offset)).await;
offset += 1000;
}
assert_eq!(query(&server, &table, vec![1005.0, 1.0]).await[0], "k1005");
server.handles.run_pending_tasks();
let session = server.reads.lock().unwrap().session.clone();
for (_, (_, dataset)) in server.handles.iter() {
assert!(Arc::ptr_eq(&dataset.session(), &session));
}
}
fn uuid_like() -> String {
format!(
"{:x}",
+471 -112
View File
@@ -8,17 +8,19 @@ use lance::Dataset;
use lance::dataset::ProjectionRequest;
use lance::dataset::WriteParams;
use lance::dataset::{MergeInsertBuilder, WhenMatched, WhenNotMatched};
use lance::io::ObjectStoreParams;
use lance::io::StorageOptionsAccessor;
use lance::io::WrappingObjectStore;
use lance::session::Session;
use lance::table::format::Fragment;
use lance_linalg::distance::DistanceType;
use prost_types::{ListValue, Struct, Value, value::Kind};
use std::collections::{BTreeMap, HashMap};
use std::sync::Arc;
use std::sync::{Arc, Mutex};
use tonic::{Request, Response, Status};
use weed_lance_worker::catalog::NamespaceClient;
use weed_lance_worker::dataset::{self, FallbackOptions};
use weed_lance_worker::catalog::TableDescription;
use weed_lance_worker::dataset;
use weed_lance_worker::direct_read::{DirectReads, Wrappers};
use weed_lance_worker::jobs::vector_index::is_vector_table;
use weed_vector_core::index;
use weed_vector_core::maintenance::SharedLeases;
use weed_vector_core::query::{self, Query};
use weed_vector_core::table::VectorTable;
@@ -35,17 +37,14 @@ fn invalid(err: impl Into<anyhow::Error>) -> Status {
fn open_status(err: anyhow::Error) -> Status {
let missing = err.chain().any(|cause| {
cause
.downcast_ref::<reqwest::Error>()
.is_some_and(|e| e.status() == Some(reqwest::StatusCode::NOT_FOUND))
|| cause.downcast_ref::<lance::Error>().is_some_and(|e| {
matches!(
e,
lance::Error::DatasetNotFound { .. }
| lance::Error::NotFound { .. }
| lance::Error::VersionNotFound { .. }
)
})
cause.downcast_ref::<lance::Error>().is_some_and(|e| {
matches!(
e,
lance::Error::DatasetNotFound { .. }
| lance::Error::NotFound { .. }
| lance::Error::VersionNotFound { .. }
)
})
});
if missing {
Status::not_found(format!("{:#}", err))
@@ -58,8 +57,9 @@ const KEY_COLUMN: &str = "key";
const METADATA_COLUMN: &str = "metadata";
const VECTOR_COLUMN: &str = "vector";
const ROW_ID_COLUMN: &str = "_rowid";
fn schema_for(
const DEFAULT_INDEX_CACHE_BYTES: usize = 1 << 30;
const DEFAULT_METADATA_CACHE_BYTES: usize = 256 << 20;
pub(crate) fn schema_for(
dimension: usize,
metric: pb::DistanceMetric,
non_filterable: &[String],
@@ -99,6 +99,44 @@ fn schema_for(
]))
}
fn matches_index_schema(
dataset: &Dataset,
dimension: usize,
metric: pb::DistanceMetric,
non_filterable: &[String],
) -> bool {
let Ok(layout) = VectorTable::resolve(dataset, Some(VECTOR_COLUMN), Some(KEY_COLUMN)) else {
return false;
};
let schema = Schema::from(dataset.schema());
let Ok(vector) = schema.field_with_name(VECTOR_COLUMN) else {
return false;
};
let Ok(metadata) = schema.field_with_name(METADATA_COLUMN) else {
return false;
};
let stored_keys = metadata
.metadata()
.get("non_filterable_metadata_keys")
.map(|keys| serde_json::from_str::<Vec<String>>(keys))
.transpose();
let Ok(stored_keys) = stored_keys else {
return false;
};
let mut stored_keys = stored_keys.unwrap_or_default();
let mut requested_keys = non_filterable.to_vec();
stored_keys.sort();
requested_keys.sort();
layout.dimension == dimension
&& vector.metadata().get("distance_metric").map(String::as_str)
== Some(metric.as_str_name())
&& stored_keys == requested_keys
&& metadata
.metadata()
.get("ARROW:extension:name")
.is_some_and(|name| name == "lance.json")
}
fn batch_of(vectors: &[pb::Vector], dimension: usize, schema: Arc<Schema>) -> Result<RecordBatch> {
let mut seen = std::collections::HashSet::new();
for vector in vectors {
@@ -370,38 +408,294 @@ fn rows_as_vectors(
out
}
/// Refuses a table of an SSE-S3 bucket when the engine's storage has no key
/// to read or write it with, rather than failing inside Lance.
fn sse_s3_ready(sse_s3: bool, key_configured: Option<bool>) -> Result<(), Status> {
if sse_s3 && key_configured == Some(false) {
return Err(Status::failed_precondition(
"the bucket encrypts with SSE-S3, which needs s3.sse.kek or s3.sse.key for the \
engine as for the S3 gateway",
));
}
Ok(())
}
/// The description a gateway sent with the table. Only a vector index at its
/// colocated path is taken, since the engine is that path's writer already;
/// any other location would point the engine at storage the catalog does not
/// know.
fn resolved(table: &pb::Table, schemes: &[&str]) -> Option<TableDescription> {
// Each id part is one path component; `..` would resolve the colocated
// path elsewhere on the filer.
if !table
.id
.iter()
.all(|part| !part.is_empty() && part != "." && part != ".." && !part.contains('/'))
{
return None;
}
let colocated = dataset::table_location(&table.id)?;
let path = colocated.strip_prefix("s3://")?;
let location = table.location.trim_end_matches('/');
let trusted = is_vector_table(&table.id)
&& schemes
.iter()
.any(|scheme| location == format!("{scheme}://{path}"));
trusted.then(|| TableDescription {
location: location.to_string(),
storage_options: dataset::unsigned_options(),
managed_versioning: false,
})
}
/// Marks the table an index was created as, in its schema metadata, which
/// every later version keeps.
const TABLE_ID_KEY: &str = "weed_vector_table_id";
/// The table a dataset was created as; empty for one created before tables
/// were marked.
pub(crate) fn table_id(dataset: &Dataset) -> &str {
dataset
.schema()
.metadata
.get(TABLE_ID_KEY)
.map_or("", String::as_str)
}
/// The session reads share, and the table each location held when this
/// session first read it. Lance keys some of its caches by location and
/// version alone, and an index dropped and created again at its location
/// numbers its versions from 1 again, so a session that sees a location hold
/// another table starts over.
pub(crate) struct Reads {
pub(crate) session: Arc<Session>,
generation: u64,
tables: HashMap<String, String>,
}
/// Opened tables kept for the next read of them.
const MAX_HANDLES: u64 = 1024;
fn handle_key(table: &pb::Table) -> (String, bool) {
(table.location.clone(), table.sse_s3)
}
/// Locations a session remembers before it starts over.
const MAX_READ_TABLES: usize = 100_000;
impl Reads {
fn new(index_bytes: usize, metadata_bytes: usize) -> Self {
Self {
session: Arc::new(Session::new(
index_bytes,
metadata_bytes,
Default::default(),
)),
generation: 0,
tables: HashMap::new(),
}
}
/// Whether a dataset of `table_id` at `location`, opened in this session,
/// may be read. Otherwise the session was replaced and the caller opens
/// the table again.
fn admit(&mut self, location: &str, table_id: &str, (index, metadata): (usize, usize)) -> bool {
match self.tables.get(location) {
Some(seen) if seen == table_id => return true,
None if self.tables.len() < MAX_READ_TABLES => {
self.tables
.insert(location.to_string(), table_id.to_string());
return true;
}
_ => {}
}
let generation = self.generation + 1;
*self = Self::new(index, metadata);
self.generation = generation;
false
}
}
pub struct VectorServer {
client: NamespaceClient,
fallback: FallbackOptions,
read_cache: Option<Arc<dyn WrappingObjectStore>>,
storage: Option<DirectReads>,
read_cache: Option<ReadCache>,
/// Lance's index and metadata caches, shared by every request that only
/// reads. Writes open in a session of their own.
pub(crate) reads: Mutex<Reads>,
/// A table opened by an earlier read, by location and encryption, with
/// the generation of the session it was opened in.
pub(crate) handles: moka::sync::Cache<(String, bool), (u64, Dataset)>,
lance_cache_bytes: (usize, usize),
leases: SharedLeases,
nprobes: Option<usize>,
/// URI schemes a table's location may have.
schemes: &'static [&'static str],
}
impl Default for VectorServer {
fn default() -> Self {
Self::new()
}
}
impl VectorServer {
pub fn new(client: NamespaceClient, fallback: FallbackOptions) -> Self {
pub fn new() -> Self {
Self {
client,
fallback,
storage: None,
read_cache: None,
reads: Mutex::new(Reads::new(
DEFAULT_INDEX_CACHE_BYTES,
DEFAULT_METADATA_CACHE_BYTES,
)),
lance_cache_bytes: (DEFAULT_INDEX_CACHE_BYTES, DEFAULT_METADATA_CACHE_BYTES),
handles: moka::sync::Cache::new(MAX_HANDLES),
leases: SharedLeases::default(),
nprobes: None,
schemes: &["s3"],
}
}
/// Tables on an in-process store, for tests.
#[cfg(test)]
pub(crate) fn with_schemes(mut self, schemes: &'static [&'static str]) -> Self {
self.schemes = schemes;
self
}
/// Partitions every query probes, instead of the index's own default.
pub fn with_nprobes(mut self, nprobes: Option<usize>) -> Self {
self.nprobes = nprobes;
self
}
/// Keep tables on the filers and volume servers, not through the gateway.
/// Unset only for tests on an in-process store.
pub fn with_storage(mut self, storage: DirectReads) -> Self {
self.storage = Some(storage);
self
}
/// Bytes of loaded indexes and of table metadata kept across reads.
pub fn with_lance_caches(mut self, index_bytes: usize, metadata_bytes: usize) -> Self {
self.reads = Mutex::new(Reads::new(index_bytes, metadata_bytes));
self.lance_cache_bytes = (index_bytes, metadata_bytes);
self
}
/// Serve repeated reads of write-once table files from `cache`.
pub fn with_read_cache(mut self, cache: ReadCache) -> Self {
self.read_cache = Some(Arc::new(cache));
self.read_cache = Some(cache);
self
}
async fn open(&self, table: &pb::Table) -> Result<Dataset> {
let opened = dataset::open_with(
&self.client,
&table.id,
&self.fallback,
self.read_cache.clone(),
)
.await?;
let mut dataset = opened.dataset;
/// The table as the gateway resolved it. A request that names no
/// location, another location than the index's own, or storage options
/// the engine's credentials must not be used with is refused.
fn describe(&self, table: &pb::Table) -> Result<TableDescription, Status> {
if table.location.is_empty() {
return Err(Status::invalid_argument("the table names no location"));
}
let description = resolved(table, self.schemes).ok_or_else(|| {
Status::permission_denied(format!(
"{} is not the location of index {}",
table.location,
table.id.join("/")
))
})?;
sse_s3_ready(
table.sse_s3,
self.storage.as_ref().map(|s| s.sse_s3_configured()),
)?;
Ok(description)
}
/// The store Lance opens a table with: direct storage innermost, the read
/// cache outside it so a hit never reaches storage.
fn wrapper(&self, table: &pb::Table) -> Option<Arc<dyn WrappingObjectStore>> {
let mut wrappers: Vec<Arc<dyn WrappingObjectStore>> = Vec::new();
if let Some(storage) = &self.storage {
wrappers.push(Arc::new(storage.encrypting(table.sse_s3)));
}
if let Some(cache) = &self.read_cache {
wrappers.push(Arc::new(cache.clone()));
}
(!wrappers.is_empty()).then(|| Arc::new(Wrappers(wrappers)) as Arc<dyn WrappingObjectStore>)
}
async fn open(&self, table: &pb::Table) -> Result<Dataset, Status> {
self.open_in(table, None).await
}
/// [`Self::open`] in the shared session, for a request that only reads.
async fn open_to_read(&self, table: &pb::Table) -> Result<Dataset, Status> {
loop {
let (session, generation) = {
let reads = self.reads.lock().unwrap();
(reads.session.clone(), reads.generation)
};
let dataset = match self.handle(table, generation).await? {
Some(dataset) => dataset,
None => {
let dataset = self.open_in(table, Some(session)).await?;
// Under the lock, so no handle of a replaced session
// lands after the replacement cleared them.
let reads = self.reads.lock().unwrap();
if table.version == 0 && reads.generation == generation {
self.handles
.insert(handle_key(table), (generation, dataset.clone()));
}
dataset
}
};
let mut reads = self.reads.lock().unwrap();
if reads.generation != generation {
continue;
}
if reads.admit(&table.location, table_id(&dataset), self.lance_cache_bytes) {
return Ok(dataset);
}
// The handles hold the replaced session and its caches.
self.handles.invalidate_all();
}
}
/// The table from the handle an earlier read of it left in this session,
/// moved to the version asked for: the latest costs a listing of its
/// versions, not a new store and manifest read.
async fn handle(&self, table: &pb::Table, generation: u64) -> Result<Option<Dataset>, Status> {
self.describe(table)?;
let key = handle_key(table);
let Some((at, mut dataset)) = self.handles.get(&key) else {
return Ok(None);
};
if at != generation {
return Ok(None);
}
let moved = match table.version {
0 => dataset.checkout_latest().await.map(|()| dataset),
version => dataset.checkout_version(version).await,
};
moved.map(Some).map_err(|err| {
self.handles.invalidate(&key);
open_status(err.into())
})
}
async fn open_in(
&self,
table: &pb::Table,
session: Option<Arc<Session>>,
) -> Result<Dataset, Status> {
let description = self.describe(table)?;
let wrapper = self.wrapper(table);
let mut dataset = dataset::open_at(&description, wrapper, session)
.await
.map_err(open_status)?;
if table.version != 0 {
dataset = dataset.checkout_version(table.version).await?;
dataset = dataset
.checkout_version(table.version)
.await
.map_err(|err| open_status(err.into()))?;
}
Ok(dataset)
}
@@ -425,30 +719,47 @@ impl VectorService for VectorServer {
return Err(Status::invalid_argument("dimension must be 1..=4096"));
}
// The gateway has already created the table through the catalog.
let description = self
.client
.describe_table(&table.id)
.await
.map_err(internal)?;
let options = dataset::table_options(&description, &self.fallback);
let description = self.describe(&table)?;
let schema = schema_for(dimension, metric, &req.non_filterable_metadata_keys);
let schema = Arc::new(Schema::new_with_metadata(
schema.fields().clone(),
HashMap::from([(
TABLE_ID_KEY.to_string(),
uuid::Uuid::new_v4().simple().to_string(),
)]),
));
let empty = RecordBatch::new_empty(schema.clone());
let reader = arrow_array::RecordBatchIterator::new(vec![Ok(empty)].into_iter(), schema);
let dataset = Dataset::write(
reader,
&description.location,
Some(WriteParams {
store_params: Some(ObjectStoreParams {
storage_options_accessor: Some(Arc::new(
StorageOptionsAccessor::with_static_options(options),
)),
..Default::default()
}),
store_params: Some(dataset::store_params(self.wrapper(&table))),
..Default::default()
}),
)
.await
.map_err(internal)?;
.await;
let dataset = match dataset {
Ok(dataset) => dataset,
Err(error) => match self.open(&table).await {
Ok(existing)
if matches_index_schema(
&existing,
dimension,
metric,
&req.non_filterable_metadata_keys,
) =>
{
existing
}
Ok(_) => {
return Err(Status::already_exists(
"index exists with a different schema",
));
}
Err(_) => return Err(internal(error)),
},
};
Ok(Response::new(pb::CreateIndexResponse {
version: dataset.version().version,
}))
@@ -462,15 +773,24 @@ impl VectorService for VectorServer {
let table = req
.table
.ok_or_else(|| Status::invalid_argument("missing table"))?;
dataset::drop(&self.client, &table.id, &self.fallback)
.await
.map_err(|err| {
if err.is::<dataset::DropRefused>() {
Status::failed_precondition(format!("{err:#}"))
} else {
internal(err)
}
})?;
// describe() already refused any location but the index's own.
let description = self.describe(&table)?;
match &self.storage {
Some(storage) => storage
.remove_table(&description.location)
.await
.map_err(internal)?,
None => {
let (store, path) = lance::io::ObjectStore::from_uri_and_params(
Arc::new(lance::io::ObjectStoreRegistry::default()),
&description.location,
&dataset::store_params(self.wrapper(&table)),
)
.await
.map_err(internal)?;
store.remove_dir_all(path).await.map_err(internal)?;
}
}
Ok(Response::new(pb::DropIndexResponse {}))
}
@@ -482,7 +802,7 @@ impl VectorService for VectorServer {
let table = req
.table
.ok_or_else(|| Status::invalid_argument("missing table"))?;
let dataset = self.open(&table).await.map_err(open_status)?;
let dataset = self.open(&table).await?;
if req.vectors.is_empty() {
return Ok(Response::new(pb::PutVectorsResponse {
version: dataset.version().version,
@@ -524,12 +844,10 @@ impl VectorService for VectorServer {
let table = req
.table
.ok_or_else(|| Status::invalid_argument("missing table"))?;
let dataset = self.open(&table).await.map_err(open_status)?;
let dataset = self.open_to_read(&table).await?;
// Lease the resolved version so maintenance cannot reclaim its files
// while this query runs against it.
let _lease = weed_vector_core::maintenance::try_pin(&dataset)
.await
.map_err(internal)?;
let _lease = self.leases.try_pin(&dataset).await.map_err(internal)?;
let resolved = VectorTable::resolve(&dataset, None, Some(KEY_COLUMN)).map_err(invalid)?;
let columns = projection(req.return_data, req.return_metadata);
let batches = query::get(&dataset, &resolved, &req.keys, Some(&columns))
@@ -548,7 +866,7 @@ impl VectorService for VectorServer {
let table = req
.table
.ok_or_else(|| Status::invalid_argument("missing table"))?;
let mut dataset = self.open(&table).await.map_err(open_status)?;
let mut dataset = self.open(&table).await?;
if !req.keys.is_empty() {
dataset
.delete(&key_list(&req.keys))
@@ -574,7 +892,7 @@ impl VectorService for VectorServer {
"segment_index must be below segment_count, which is at most {MAX_SEGMENTS}"
)));
}
let dataset = self.open(&table).await.map_err(open_status)?;
let dataset = self.open_to_read(&table).await?;
let offset: u64 = if req.next_token.is_empty() {
0
} else {
@@ -594,9 +912,7 @@ impl VectorService for VectorServer {
// Data and metadata come from a second read of this version, so
// lease it as get_vectors does.
let _lease = if columns.len() > 1 {
weed_vector_core::maintenance::try_pin(&dataset)
.await
.map_err(internal)?
self.leases.try_pin(&dataset).await.map_err(internal)?
} else {
None
};
@@ -641,12 +957,10 @@ impl VectorService for VectorServer {
let table = req
.table
.ok_or_else(|| Status::invalid_argument("missing table"))?;
let dataset = self.open(&table).await.map_err(open_status)?;
let dataset = self.open_to_read(&table).await?;
// Lease the resolved version so maintenance cannot reclaim its files
// while this query runs against it.
let _lease = weed_vector_core::maintenance::try_pin(&dataset)
.await
.map_err(internal)?;
let _lease = self.leases.try_pin(&dataset).await.map_err(internal)?;
let resolved = VectorTable::resolve(&dataset, None, Some(KEY_COLUMN)).map_err(invalid)?;
if req.query_vector.len() != resolved.dimension {
return Err(Status::invalid_argument(format!(
@@ -658,16 +972,7 @@ impl VectorService for VectorServer {
if req.top_k == 0 {
return Err(Status::invalid_argument("top_k must be positive"));
}
// The index's configured metric lives in the vector field's metadata.
let distance = match dataset
.schema()
.field(VECTOR_COLUMN)
.and_then(|f| f.metadata.get("distance_metric").map(String::as_str))
{
Some("COSINE") => DistanceType::Cosine,
Some("DOT_PRODUCT") => DistanceType::Dot,
_ => DistanceType::L2,
};
let distance = index::distance_metric(&dataset, VECTOR_COLUMN);
let query = Query {
vector: req.query_vector,
k: req.top_k as usize,
@@ -678,7 +983,7 @@ impl VectorService for VectorServer {
},
postfilter: req.postfilter,
distance,
nprobes: None,
nprobes: self.nprobes,
index_segments: None,
fragments: None,
};
@@ -942,45 +1247,99 @@ mod tests {
assert_eq!(next, None);
}
/// A catalog that answers one describe with `location`.
async fn catalog_describing(location: &str) -> String {
use tokio::io::{AsyncReadExt, AsyncWriteExt};
let listener = tokio::net::TcpListener::bind("127.0.0.1:0").await.unwrap();
let url = format!("http://{}", listener.local_addr().unwrap());
let body = serde_json::json!({ "location": location }).to_string();
tokio::spawn(async move {
let (mut conn, _) = listener.accept().await.unwrap();
let mut request = [0u8; 8192];
let _ = conn.read(&mut request).await;
let response = format!(
"HTTP/1.1 200 OK\r\ncontent-type: application/json\r\ncontent-length: {}\r\nconnection: close\r\n\r\n{body}",
body.len()
);
conn.write_all(response.as_bytes()).await.unwrap();
let _ = conn.shutdown().await;
let _ = tokio::io::copy(&mut conn, &mut tokio::io::sink()).await;
});
url
}
/// A location that is not the index's own is refused before anything is
/// opened or removed.
#[tokio::test]
async fn drop_index_refuses_a_location_outside_the_table() {
for location in ["s3://bkt/vector", "s3://bkt/vector/prod"] {
let server = VectorServer::new(
NamespaceClient::new(catalog_describing(location).await),
FallbackOptions::new(),
);
let server = VectorServer::new();
for location in [
"s3://bkt/vector",
"s3://bkt/vector/prod",
"s3://other/vector/x",
] {
let request = pb::DropIndexRequest {
table: Some(pb::Table {
id: vec!["bkt".into(), "vector".into(), "x".into()],
version: 0,
location: location.into(),
..Default::default()
}),
};
let err = server
.drop_index(Request::new(request))
.await
.expect_err(location);
assert_eq!(err.code(), tonic::Code::FailedPrecondition, "{location}");
assert_eq!(err.code(), tonic::Code::PermissionDenied, "{location}");
}
}
/// A table of an SSE-S3 bucket is refused before Lance opens it when the
/// storage holds no key.
#[test]
fn an_sse_s3_table_needs_the_key() {
assert!(sse_s3_ready(false, Some(false)).is_ok());
assert!(sse_s3_ready(true, Some(true)).is_ok());
assert!(
sse_s3_ready(true, None).is_ok(),
"in-process stores need no key"
);
let err = sse_s3_ready(true, Some(false)).unwrap_err();
assert_eq!(err.code(), tonic::Code::FailedPrecondition);
assert!(err.message().contains("s3.sse.kek or s3.sse.key"));
}
/// Only a vector index at its colocated path is taken from the request.
#[test]
fn resolved_takes_only_a_colocated_vector_index() {
let table = |id: &[&str], location: &str| pb::Table {
id: id.iter().map(|s| s.to_string()).collect(),
location: location.into(),
..Default::default()
};
let s3 = &["s3"];
let index = ["bkt", "vector", "x"];
for location in ["s3://bkt/vector/x", "s3://bkt/vector/x/"] {
let description = resolved(&table(&index, location), s3).expect(location);
assert_eq!(description.location, "s3://bkt/vector/x");
assert_eq!(
description
.storage_options
.get("aws_skip_signature")
.map(String::as_str),
Some("true")
);
}
assert!(resolved(&table(&index, ""), s3).is_none());
assert!(resolved(&table(&index, "s3://other/vector/x"), s3).is_none());
assert!(resolved(&table(&index, "s3://bkt/vector/y"), s3).is_none());
assert!(resolved(&table(&index, "file://bkt/vector/x"), s3).is_none());
assert!(resolved(&table(&index, "memory://bkt/vector/x"), &["s3", "memory"]).is_some());
assert!(resolved(&table(&["bkt", "ns", "x"], "s3://bkt/ns/x"), s3).is_none());
assert!(resolved(&table(&["bkt", "vector", ".."], "s3://bkt/vector/.."), s3).is_none());
assert!(resolved(&table(&["bkt", "vector", "a/b"], "s3://bkt/vector/a/b"), s3).is_none());
assert!(resolved(&table(&["bkt", "x"], "s3://bkt/x"), s3).is_none());
}
/// A request names the table's location or is refused.
#[tokio::test]
async fn a_table_without_its_location_is_refused() {
let server = VectorServer::new();
for (location, code) in [
("", tonic::Code::InvalidArgument),
("s3://other/vector/x", tonic::Code::PermissionDenied),
] {
let err = server
.get_vectors(Request::new(pb::GetVectorsRequest {
table: Some(pb::Table {
id: vec!["bkt".into(), "vector".into(), "x".into()],
location: location.into(),
..Default::default()
}),
keys: vec!["k001".into()],
..Default::default()
}))
.await
.expect_err(location);
assert_eq!(err.code(), code, "{location}");
}
}
}
@@ -0,0 +1,111 @@
use std::sync::Arc;
use bytes::Bytes;
use lance::io::WrappingObjectStore;
use object_store::{ObjectStoreExt, memory::InMemory, path::Path};
use weed_vector_server::{metrics::Metrics, read_cache::ReadCache};
#[tokio::test]
async fn immutable_reads_report_hits_misses_bytes_and_weight() {
let metrics = Metrics::new().unwrap();
let cache = ReadCache::new(1024).with_metrics(metrics.clone());
let store = cache.wrap("memory", Arc::new(InMemory::new()));
let path = Path::from("data/example.lance");
store
.put(&path, Bytes::from_static(b"abc").into())
.await
.unwrap();
for _ in 0..2 {
assert_eq!(
store.get(&path).await.unwrap().bytes().await.unwrap(),
"abc"
);
}
assert_eq!(metrics.cache_requests.with_label_values(&["miss"]).get(), 1);
assert_eq!(metrics.cache_requests.with_label_values(&["hit"]).get(), 1);
assert_eq!(metrics.storage_bytes.get(), 3);
assert_eq!(metrics.storage_read_seconds.get_sample_count(), 1);
let (entries, weight) = cache.size();
assert_eq!(entries, 1);
assert!(weight >= 3);
let metrics_endpoint = tokio::net::TcpListener::bind("127.0.0.1:0").await.unwrap();
let endpoint = format!("http://{}/metrics", metrics_endpoint.local_addr().unwrap());
let task = tokio::spawn(weed_vector_server::metrics::serve_listener(
metrics,
metrics_endpoint,
Some(cache),
));
let body = reqwest::get(endpoint).await.unwrap().text().await.unwrap();
assert!(body.contains("SeaweedFS_vector_cache_entries 1"));
assert!(body.contains("SeaweedFS_vector_cache_bytes "));
task.abort();
}
#[tokio::test]
async fn cancelled_requests_are_counted_once() {
let metrics = Metrics::new().unwrap();
let guard = metrics.start("QueryVectors");
let task = tokio::spawn(async move {
let _guard = guard;
std::future::pending::<()>().await;
});
task.abort();
let _ = task.await;
assert_eq!(
metrics.in_flight.with_label_values(&["QueryVectors"]).get(),
0
);
assert_eq!(
metrics
.requests
.with_label_values(&["QueryVectors", "Cancelled"])
.get(),
1
);
assert_eq!(
metrics
.request_seconds
.with_label_values(&["QueryVectors"])
.get_sample_count(),
1
);
{
let mut guard = metrics.start("QueryVectors");
guard.complete(None);
}
assert_eq!(
metrics
.requests
.with_label_values(&["QueryVectors", "OK"])
.get(),
1
);
assert_eq!(
metrics
.requests
.with_label_values(&["QueryVectors", "Cancelled"])
.get(),
1
);
}
#[tokio::test]
async fn large_immutable_reads_report_bytes_without_caching() {
let metrics = Metrics::new().unwrap();
let cache = ReadCache::new(16 * 1024 * 1024).with_metrics(metrics.clone());
let store = cache.wrap("memory", Arc::new(InMemory::new()));
let path = Path::from("data/large.lance");
let size = 5 * 1024 * 1024;
store
.put(&path, Bytes::from(vec![0; size]).into())
.await
.unwrap();
let result = store.get(&path).await.unwrap();
assert_eq!(metrics.storage_bytes.get(), 0);
assert_eq!(metrics.storage_read_seconds.get_sample_count(), 0);
assert_eq!(result.bytes().await.unwrap().len(), size);
assert_eq!(metrics.storage_bytes.get(), size as u64);
assert_eq!(metrics.cache_requests.with_label_values(&["miss"]).get(), 1);
assert_eq!(metrics.storage_read_seconds.get_sample_count(), 1);
assert_eq!(cache.size(), (0, 0));
}
+1
View File
@@ -4608,6 +4608,7 @@ dependencies = [
"jsonwebtoken",
"kamadak-exif",
"libc",
"lru",
"md-5",
"memmap2",
"mime_guess",
+1
View File
@@ -130,6 +130,7 @@ chrono = "0.4"
hex = "0.4"
parking_lot = "0.12"
dashmap = "6"
lru = "0.16"
thiserror = "1"
anyhow = "1"
async-trait = "0.1"
+4
View File
@@ -230,6 +230,10 @@ message KeepConnectedRequest {
// A draining filer leaves the lock ring but stays a cluster member, so peers
// keep following its metadata until the stream closes.
bool leave_lock_ring = 8;
// Port of this node's Prometheus /metrics listener (-metricsPort), or 0 when
// it is not enabled. Advertised so the admin server can scrape it without
// exposing metrics on the client-facing service port.
uint32 metrics_port = 9;
}
message VolumeLocation {
+48 -35
View File
@@ -1,7 +1,7 @@
use clap::Parser;
use std::ffi::OsString;
use std::net::UdpSocket;
use std::path::{Path, PathBuf};
use std::path::PathBuf;
use crate::security::tls::TlsPolicy;
@@ -1051,8 +1051,6 @@ pub struct SecurityConfig {
pub guard_white_list: Vec<String>,
}
const SECURITY_CONFIG_FILE_NAME: &str = "security.toml";
/// Parse a security.toml file to extract JWT signing keys and TLS configuration.
/// Format:
/// ```toml
@@ -1177,7 +1175,7 @@ fn parse_security_config_with_env(path: &str, env: EnvLookup<'_>) -> SecurityCon
if let Some((key, value)) = trimmed.split_once('=') {
let key = key.trim();
let value = value.trim().trim_matches('"');
let value: &str = &toml_value_text(value);
match section {
Section::JwtSigningRead => match key {
"key" => cfg.jwt_read_signing_key = value.as_bytes().to_vec(),
@@ -1267,44 +1265,29 @@ fn parse_security_config_with_env(path: &str, env: EnvLookup<'_>) -> SecurityCon
cfg
}
/// Reads the right-hand side of `key = value` the way TOML (and so Go) does:
/// escapes decoded, literal strings kept as written, a trailing comment
/// dropped. Text TOML rejects keeps the plain quote trimming.
fn toml_value_text(raw: &str) -> String {
match format!("v = {raw}").parse::<toml::Table>() {
Ok(mut table) => match table.remove("v") {
Some(toml::Value::String(text)) => text,
Some(other) => other.to_string(),
None => String::new(),
},
Err(_) => raw.trim().trim_matches('"').to_string(),
}
}
fn resolve_security_config_path(path: &str, env: EnvLookup<'_>) -> Option<PathBuf> {
if !path.is_empty() {
return Some(PathBuf::from(path));
}
default_security_config_candidates(
seaweed_common::security_config::find(
std::env::current_dir().ok().as_deref(),
home_dir_from_env(env).as_deref(),
seaweed_common::security_config::home_dir(env).as_deref(),
)
.into_iter()
.find(|candidate| candidate.is_file())
}
fn default_security_config_candidates(
current_dir: Option<&Path>,
home_dir: Option<&Path>,
) -> Vec<PathBuf> {
let mut candidates = Vec::new();
if let Some(dir) = current_dir {
candidates.push(dir.join(SECURITY_CONFIG_FILE_NAME));
}
if let Some(home) = home_dir {
candidates.push(home.join(".seaweedfs").join(SECURITY_CONFIG_FILE_NAME));
}
candidates.push(PathBuf::from("/usr/local/etc/seaweedfs").join(SECURITY_CONFIG_FILE_NAME));
candidates.push(PathBuf::from("/etc/seaweedfs").join(SECURITY_CONFIG_FILE_NAME));
candidates
}
fn home_dir_from_env(env: EnvLookup<'_>) -> Option<PathBuf> {
env("HOME")
.filter(|v| !v.is_empty())
.map(PathBuf::from)
.or_else(|| {
env("USERPROFILE")
.filter(|v| !v.is_empty())
.map(PathBuf::from)
})
}
/// Apply WEED_ environment variable overrides to a SecurityConfig.
@@ -1403,7 +1386,9 @@ fn detect_host_address() -> String {
#[cfg(test)]
mod tests {
use super::*;
use seaweed_common::security_config::SECURITY_CONFIG_FILE_NAME;
use std::collections::HashMap;
use std::path::Path;
use std::sync::{Mutex, MutexGuard, OnceLock};
/// Serialises the tests that move the process working directory against
@@ -1897,6 +1882,34 @@ ui = true
assert!(cfg.access_ui);
}
#[test]
fn test_parse_security_config_reads_toml_strings() {
for (encoded, expected) in [
(r#""plain""#, "plain"),
(r#""key\n""#, "key\n"),
(r#""quote\"slash\\tab\t""#, "quote\"slash\\tab\t"),
(r#""key""#, "key"),
("'literal\\key'", "literal\\key"),
(r#""key" # comment"#, "key"),
(r#""YWJj==""#, "YWJj=="),
("unquoted", "unquoted"),
] {
let tmp = tempfile::NamedTempFile::new().unwrap();
std::fs::write(
tmp.path(),
format!(
"[jwt.signing]\nkey = {encoded}\n\n[jwt.signing.read]\nkey = {encoded}\n\
expires_after_seconds = 30 # seconds\n"
),
)
.unwrap();
let cfg = parse_security_config_with_env(tmp.path().to_str().unwrap(), &|_| None);
assert_eq!(cfg.jwt_signing_key, expected.as_bytes(), "{encoded}");
assert_eq!(cfg.jwt_read_signing_key, expected.as_bytes(), "{encoded}");
assert_eq!(cfg.jwt_read_signing_expires, 30, "{encoded}");
}
}
#[test]
fn test_parse_security_config_discovers_current_directory_default() {
let _guard = process_state_lock();
+3 -3
View File
@@ -143,12 +143,12 @@ fn flip_horizontal_if(img: DynamicImage, do_flip: bool) -> DynamicImage {
DynamicImage::ImageRgba8(out)
}
/// Returns true if the given MIME type or file path extension indicates a JPEG file.
pub fn is_jpeg(mime_type: &str, path: &str) -> bool {
/// Returns true if the given MIME type or file name extension indicates a JPEG file.
pub fn is_jpeg(mime_type: &str, filename: &str) -> bool {
if mime_type == "image/jpeg" {
return true;
}
let lower = path.to_lowercase();
let lower = filename.to_lowercase();
lower.ends_with(".jpg") || lower.ends_with(".jpeg")
}
+185 -35
View File
@@ -118,6 +118,17 @@ fn reject_https_downgrade(attempt: reqwest::redirect::Attempt) -> reqwest::redir
}
}
/// A peer that never answers a SYN (silent partition) must not stall new
/// outgoing connections; requests set their own overall bounds as needed.
const OUTGOING_CONNECT_TIMEOUT: std::time::Duration = std::time::Duration::from_secs(5);
/// Base of every outgoing HTTP client: redirect policy and connect bound.
fn outgoing_client_builder(redirect_policy: reqwest::redirect::Policy) -> reqwest::ClientBuilder {
reqwest::Client::builder()
.redirect(redirect_policy)
.connect_timeout(OUTGOING_CONNECT_TIMEOUT)
}
fn build_outgoing_http_client(
config: &VolumeServerConfig,
) -> Result<(reqwest::Client, String), Box<dyn std::error::Error>> {
@@ -129,14 +140,12 @@ fn build_outgoing_http_client(
let redirect_policy = reqwest::redirect::Policy::custom(reject_https_downgrade);
if !config.https_client_enabled {
return Ok((
reqwest::Client::builder()
.redirect(redirect_policy)
.build()?,
outgoing_client_builder(redirect_policy).build()?,
scheme.to_string(),
));
}
let mut builder = reqwest::Client::builder().redirect(redirect_policy);
let mut builder = outgoing_client_builder(redirect_policy);
if !config.https_client_ca_file.is_empty() {
let ca_pem = std::fs::read(&config.https_client_ca_file).map_err(|e| {
format!(
@@ -708,7 +717,8 @@ async fn run(
.build_v1alpha()
.expect("Failed to build gRPC reflection v1alpha service");
let result = if let Some(tls_acceptor) = grpc_tls_acceptor {
let incoming = grpc_tls_incoming(grpc_listener, tls_acceptor);
let incoming =
grpc_tls_incoming(grpc_listener, tls_acceptor, GRPC_TLS_HANDSHAKE_TIMEOUT);
info!("gRPC server listening on {} (TLS enabled)", grpc_local_addr);
build_grpc_server_builder()
.layer(GrpcRequestIdLayer)
@@ -845,40 +855,39 @@ async fn run(
}))
};
// Wait for servers. Use select! with &mut so the losing handle is not
// dropped, then await it explicitly afterward.
// Whichever server stops first takes the other down, as a serve failure
// is fatal for the Go volume server.
let mut server_err: Option<String> = None;
let mut http_handle = http_handle;
let mut grpc_handle = grpc_handle;
let grpc_finished_first = tokio::select! {
_ = &mut http_handle => false,
_ = &mut grpc_handle => true,
};
// Inspect the gRPC result (already resolved if it finished first,
// otherwise await it now).
let grpc_result = if grpc_finished_first {
grpc_handle.await
} else {
// HTTP finished first; gRPC is still running. Await it.
grpc_handle.await
let mut http_handle = http_handle;
let (http_result, grpc_result) = tokio::select! {
r = &mut grpc_handle => {
let _ = shutdown_tx.send(());
(http_handle.await, r)
}
r = &mut http_handle => {
let _ = shutdown_tx.send(());
(r, grpc_handle.await)
}
};
if let Err(e) = http_result {
let msg = format!("HTTP task panicked: {}", e);
error!("{}", msg);
server_err = Some(msg);
}
match grpc_result {
Ok(Ok(())) => {}
Ok(Err(e)) => {
let msg = format!("gRPC server exited with error: {}", e);
error!("{}", msg);
server_err = Some(msg);
// serve error already sent shutdown inside the task
}
Err(e) => {
let msg = format!("gRPC task panicked: {}", e);
error!("{}", msg);
server_err = Some(msg);
let _ = shutdown_tx.send(());
}
}
// Ensure the HTTP handle completes too.
let _ = http_handle.await;
if let Some(h) = public_handle {
let _ = h.await;
}
@@ -946,28 +955,64 @@ async fn run_metrics_push_loop(
}
}
/// grpc-go's default ConnectionTimeout, which bounds a Go server's handshake.
const GRPC_TLS_HANDSHAKE_TIMEOUT: std::time::Duration = std::time::Duration::from_secs(120);
/// Each handshake runs on its own task, so a peer that connects and stays
/// silent holds up no other connection, as with a Go gRPC server.
fn grpc_tls_incoming(
listener: tokio::net::TcpListener,
tls_acceptor: TlsAcceptor,
handshake_timeout: std::time::Duration,
) -> impl tokio_stream::Stream<
Item = Result<tokio_rustls::server::TlsStream<tokio::net::TcpStream>, std::io::Error>,
> {
async_stream::stream! {
let (tx, rx) = tokio::sync::mpsc::channel(64);
tokio::spawn(async move {
let mut retry_delay = std::time::Duration::ZERO;
loop {
match listener.accept().await {
Ok((tcp_stream, remote_addr)) => match tls_acceptor.accept(tcp_stream).await {
Ok(tls_stream) => yield Ok(tls_stream),
Err(e) => {
let accepted = tokio::select! {
_ = tx.closed() => return,
accepted = listener.accept() => accepted,
};
let (tcp_stream, remote_addr) = match accepted {
Ok(accepted) => {
retry_delay = std::time::Duration::ZERO;
accepted
}
Err(e) => {
// Back off and retry like grpc-go, so running out of file
// descriptors does not close the gRPC port for good.
retry_delay = (retry_delay * 2).clamp(
std::time::Duration::from_millis(5),
std::time::Duration::from_secs(1),
);
tracing::warn!("gRPC accept error: {}; retrying in {:?}", e, retry_delay);
tokio::select! {
_ = tx.closed() => return,
_ = tokio::time::sleep(retry_delay) => continue,
}
}
};
let tls_acceptor = tls_acceptor.clone();
let tx = tx.clone();
tokio::spawn(async move {
match tokio::time::timeout(handshake_timeout, tls_acceptor.accept(tcp_stream)).await
{
Ok(Ok(tls_stream)) => {
let _ = tx.send(Ok(tls_stream)).await;
}
Ok(Err(e)) => {
tracing::debug!("gRPC TLS handshake failed from {}: {}", remote_addr, e);
}
},
Err(e) => {
yield Err(e);
break;
Err(_) => {
tracing::debug!("gRPC TLS handshake from {} timed out", remote_addr);
}
}
}
});
}
}
});
tokio_stream::wrappers::ReceiverStream::new(rx)
}
/// Serve an axum Router over TLS using tokio-rustls.
@@ -1074,16 +1119,121 @@ async fn serve_https<F>(
#[cfg(test)]
mod tests {
use super::{
build_grpc_server_tls_acceptor, effective_http_tls_policy, tls_policy_is_configured,
GRPC_TLS_HANDSHAKE_TIMEOUT, build_grpc_server_tls_acceptor, effective_http_tls_policy,
grpc_tls_incoming, tls_policy_is_configured,
};
use seaweed_volume::security::tls::TlsPolicy;
/// A connect that never finishes fails at the 5 s bound. It stalls in the
/// resolver, which the connect bound also covers, because that needs no
/// routing: a real unanswered SYN depends on the host's network.
#[tokio::test(start_paused = true)]
async fn outgoing_client_bounds_a_stalled_connect() {
struct Stalled;
impl reqwest::dns::Resolve for Stalled {
fn resolve(&self, _: reqwest::dns::Name) -> reqwest::dns::Resolving {
Box::pin(std::future::pending())
}
}
let client = super::outgoing_client_builder(reqwest::redirect::Policy::none())
.no_proxy()
.dns_resolver(std::sync::Arc::new(Stalled))
.build()
.unwrap();
let started = tokio::time::Instant::now();
let sent = tokio::time::timeout(
std::time::Duration::from_secs(600),
client.get("http://silent-peer.invalid:8080/status").send(),
)
.await
.expect("a stalled connect must not outlive its bound");
assert!(sent.is_err(), "a stalled connect cannot answer");
assert_eq!(started.elapsed(), super::OUTGOING_CONNECT_TIMEOUT);
}
fn write_pem(dir: &tempfile::TempDir, name: &str, body: &str) -> String {
let path = dir.path().join(name);
std::fs::write(&path, body).unwrap();
path.to_string_lossy().into_owned()
}
// A CA and a leaf it signed for localhost, usable on either side.
const TEST_CA_PEM: &str = "-----BEGIN CERTIFICATE-----\nMIIBhzCCAS2gAwIBAgIUWQMpep+B+ShfbwEfk20OGMkBgpAwCgYIKoZIzj0EAwIw\nGTEXMBUGA1UEAwwOdm9sdW1lLXRlc3QtY2EwHhcNMjYxMTAzMTgzOTI2WhcNMzYx\nMDMxMTgzOTI2WjAZMRcwFQYDVQQDDA52b2x1bWUtdGVzdC1jYTBZMBMGByqGSM49\nAgEGCCqGSM49AwEHA0IABD8L549bGTiFShKDcA2UFhz2MH9wxVuFCWCfJ7OVYM2x\nweY/5uCLjBRAnFx5FuWnaNtdTWkImXBSguHe861foCajUzBRMB0GA1UdDgQWBBQH\nOSZ93m5jmlLJpS5cFzH2hicEdjAfBgNVHSMEGDAWgBQHOSZ93m5jmlLJpS5cFzH2\nhicEdjAPBgNVHRMBAf8EBTADAQH/MAoGCCqGSM49BAMCA0gAMEUCIEetcimwxiDU\n8RS0b5uaev3c/kbnOT1nOV6n2wdtplZ4AiEAlWDH77I9ssy+yqXArkdiEadePIl5\nUyXg6crXqITwUeE=\n-----END CERTIFICATE-----\n";
const TEST_LEAF_PEM: &str = "-----BEGIN CERTIFICATE-----\nMIIBuTCCAV+gAwIBAgIUC4xhlzsyNZi0XHzyCwV/xquGQLkwCgYIKoZIzj0EAwIw\nGTEXMBUGA1UEAwwOdm9sdW1lLXRlc3QtY2EwHhcNMjYxMTAzMTgzOTI2WhcNMzYx\nMDMxMTgzOTI2WjAUMRIwEAYDVQQDDAlsb2NhbGhvc3QwWTATBgcqhkjOPQIBBggq\nhkjOPQMBBwNCAASKt0T/i1pt7Aw3DloYFRoaX8+w/reR3tIEaMo911/fIvQhqkg4\nLPCXKocq0KNw3Na3HnZeU9Ls+FLoSHynBsyIo4GJMIGGMBoGA1UdEQQTMBGCCWxv\nY2FsaG9zdIcEfwAAATAdBgNVHSUEFjAUBggrBgEFBQcDAQYIKwYBBQUHAwIwCQYD\nVR0TBAIwADAdBgNVHQ4EFgQUp8jNqBiE0UeYRkCny2In6SyKEKkwHwYDVR0jBBgw\nFoAUBzkmfd5uY5pSyaUuXBcx9oYnBHYwCgYIKoZIzj0EAwIDSAAwRQIhAIzNfGpS\n+rIxr3eGInfD+0DzQxSuqfcQZTkV2eGkebcuAiAIV73Y7wKg7D08qfmmrH/TaAZB\nm2wr/1assOVDaw8dMg==\n-----END CERTIFICATE-----\n";
const TEST_LEAF_KEY_PEM: &str = "-----BEGIN PRIVATE KEY-----\nMIGHAgEAMBMGByqGSM49AgEGCCqGSM49AwEHBG0wawIBAQQgmMs2vKPN5ymhYOXl\nztUUiSWAcmKVVWxveajzfxIwqBehRANCAASKt0T/i1pt7Aw3DloYFRoaX8+w/reR\n3tIEaMo911/fIvQhqkg4LPCXKocq0KNw3Na3HnZeU9Ls+FLoSHynBsyI\n-----END PRIVATE KEY-----\n";
fn test_grpc_tls_acceptor(dir: &tempfile::TempDir) -> tokio_rustls::TlsAcceptor {
seaweed_volume::security::tls::install_default_crypto_provider();
let ca = write_pem(dir, "ca.pem", TEST_CA_PEM);
let cert = write_pem(dir, "leaf.pem", TEST_LEAF_PEM);
let key = write_pem(dir, "leaf.key", TEST_LEAF_KEY_PEM);
build_grpc_server_tls_acceptor(&cert, &key, &ca, &TlsPolicy::default(), "", &[])
.unwrap()
.unwrap()
}
#[tokio::test]
async fn test_grpc_tls_incoming_serves_past_a_silent_peer() {
use tokio_stream::StreamExt;
let dir = tempfile::tempdir().unwrap();
let acceptor = test_grpc_tls_acceptor(&dir);
let listener = tokio::net::TcpListener::bind("127.0.0.1:0").await.unwrap();
let addr = listener.local_addr().unwrap();
let mut incoming = Box::pin(grpc_tls_incoming(
listener,
acceptor,
GRPC_TLS_HANDSHAKE_TIMEOUT,
));
let _silent = tokio::net::TcpStream::connect(addr).await.unwrap();
let mut roots = rustls::RootCertStore::empty();
for c in rustls_pemfile::certs(&mut TEST_CA_PEM.as_bytes()) {
roots.add(c.unwrap()).unwrap();
}
let config = rustls::ClientConfig::builder()
.with_root_certificates(roots)
.with_client_auth_cert(
rustls_pemfile::certs(&mut TEST_LEAF_PEM.as_bytes())
.map(|c| c.unwrap())
.collect(),
rustls_pemfile::private_key(&mut TEST_LEAF_KEY_PEM.as_bytes())
.unwrap()
.unwrap(),
)
.unwrap();
let client = tokio::spawn(async move {
let tcp = tokio::net::TcpStream::connect(addr).await.unwrap();
tokio_rustls::TlsConnector::from(std::sync::Arc::new(config))
.connect("localhost".try_into().unwrap(), tcp)
.await
});
let served = tokio::time::timeout(std::time::Duration::from_secs(5), incoming.next()).await;
assert!(
matches!(served, Ok(Some(Ok(_)))),
"the handshake waited behind a silent peer"
);
client.await.unwrap().unwrap();
}
#[tokio::test]
async fn test_grpc_tls_incoming_closes_a_silent_peer_after_the_handshake_timeout() {
use tokio::io::AsyncReadExt;
let dir = tempfile::tempdir().unwrap();
let acceptor = test_grpc_tls_acceptor(&dir);
let listener = tokio::net::TcpListener::bind("127.0.0.1:0").await.unwrap();
let addr = listener.local_addr().unwrap();
let _incoming =
grpc_tls_incoming(listener, acceptor, std::time::Duration::from_millis(100));
let mut silent = tokio::net::TcpStream::connect(addr).await.unwrap();
let mut buf = [0u8; 1];
let read =
tokio::time::timeout(std::time::Duration::from_secs(5), silent.read(&mut buf)).await;
assert!(
matches!(read, Ok(Ok(0)) | Ok(Err(_))),
"the silent peer stayed connected past the handshake timeout"
);
}
#[test]
fn test_grpc_server_tls_requires_ca() {
let dir = tempfile::tempdir().unwrap();
+16
View File
@@ -252,6 +252,21 @@ pub static SCRUB_SHARD_FAILURES: LazyLock<IntCounterVec> = LazyLock::new(|| {
.expect("metric can be created")
});
/// Reads of an EC shard interval that couldn't be read directly and were
/// served by reconstructing it from the other shards, by result. `success`
/// means the reconstruction produced data, not that the needle then passed
/// validation. Mirrors Go's VolumeServerECReconstructedIntervalCounter.
pub static EC_RECONSTRUCTED_INTERVAL_COUNTER: LazyLock<IntCounterVec> = LazyLock::new(|| {
IntCounterVec::new(
Opts::new(
"SeaweedFS_volumeServer_ec_reconstructed_interval_total",
"Reads of an EC interval served by reconstructing from other shards; success means reconstruction produced data, not that the needle passed validation.",
),
&["result"],
)
.expect("metric can be created")
});
/// Counter of storage read/write EIO errors on volumes and EC shards.
/// Mirrors Go's VolumeServerStorageIoErrorCounter.
pub static STORAGE_IO_ERROR_COUNTER: LazyLock<IntCounter> = LazyLock::new(|| {
@@ -402,6 +417,7 @@ pub fn register_metrics() {
Box::new(SCRUB_VOLUME_FAILURES.clone()),
Box::new(SCRUB_SHARD_FAILURES.clone()),
Box::new(STORAGE_IO_ERROR_COUNTER.clone()),
Box::new(EC_RECONSTRUCTED_INTERVAL_COUNTER.clone()),
Box::new(IO_QUARANTINE_GAUGE.clone()),
// Legacy metrics
Box::new(VOLUMES_TOTAL.clone()),
File diff suppressed because it is too large. Load diff
File diff suppressed because it is too large. Load diff
+340 -34
View File
@@ -64,10 +64,11 @@ pub async fn run_heartbeat_with_state(
let pulse = Duration::from_secs(config.pulse_seconds.max(1));
let mut new_leader: Option<String> = None;
let mut duplicate_retry_count: u32 = 0;
// Whether a heartbeat stream was ever answered by a leader; before first
// registration the retry loop reconnects promptly instead of pacing a
// full pulse. Mirrors Go's `confirmedLeaderOnce`.
let mut confirmed_leader_once = false;
// When a stream a leader had answered last ended, `None` before first
// registration. Until then, and for a pulse after each loss, the retry
// loop reconnects promptly instead of pacing a full pulse. Mirrors Go's
// `confirmedLeaderOnce` and `leaderLostAt`.
let mut leader_lost_at: Option<tokio::time::Instant> = None;
loop {
let mut tried_hints: HashSet<String> = HashSet::new();
@@ -110,7 +111,7 @@ pub async fn run_heartbeat_with_state(
&leader,
pulse,
&mut shutdown_rx,
&mut confirmed_leader_once,
&mut leader_lost_at,
&mut duplicate_retry_count,
)
.await;
@@ -134,7 +135,7 @@ pub async fn run_heartbeat_with_state(
master_addr,
pulse,
&mut shutdown_rx,
&mut confirmed_leader_once,
&mut leader_lost_at,
&mut duplicate_retry_count,
)
.await
@@ -168,7 +169,7 @@ async fn heartbeat_once(
target_addr: &str,
pulse: Duration,
shutdown_rx: &mut broadcast::Receiver<()>,
confirmed_leader_once: &mut bool,
leader_lost_at: &mut Option<tokio::time::Instant>,
duplicate_retry_count: &mut u32,
) -> Option<String> {
let grpc_addr = to_grpc_address(target_addr);
@@ -181,17 +182,22 @@ async fn heartbeat_once(
SleepDuplicate(Duration),
SleepPulse,
}
let action = match do_heartbeat(
let mut confirmed_leader = false;
let result = do_heartbeat(
config,
state,
&grpc_addr,
target_addr,
pulse,
shutdown_rx,
confirmed_leader_once,
&mut confirmed_leader,
)
.await
{
.map_err(|e| e.to_string());
if confirmed_leader {
*leader_lost_at = Some(tokio::time::Instant::now());
}
let action = match result {
Ok(Some(leader)) => {
info!("Master leader changed to {}", leader);
return Some(leader);
@@ -200,10 +206,7 @@ async fn heartbeat_once(
*duplicate_retry_count = 0;
return None;
}
Err(e) => {
let err_msg = e.to_string();
// Drop `e` (non-Send) before any .await
drop(e);
Err(err_msg) => {
warn!("Heartbeat to {} error: {}", grpc_addr, err_msg);
if err_msg.contains(DUPLICATE_UUID_RETRY_MESSAGE) {
@@ -236,16 +239,17 @@ async fn heartbeat_once(
// The lookup spends part of the retry delay rather than adding
// to it.
let retry_at = tokio::time::Instant::now() + pulse;
let unregistered_or_just_lost = leader_lost_at.is_none_or(|at| at.elapsed() < pulse);
let leader = refresh_leader_from_peers(config, state, target_addr, pulse).await;
let mut wait = retry_at.saturating_duration_since(tokio::time::Instant::now());
if wait > Duration::from_secs(1)
&& (!state.current_master().await.is_empty() || !*confirmed_leader_once)
&& (!state.current_master().await.is_empty() || unregistered_or_just_lost)
{
// A leader just named by the refresh usually accepts the
// next stream right away, and an unregistered server has
// nothing else to do, so reconnect promptly. Once the
// server was registered and no leader is known, keep the
// pulse backoff.
// next stream right away, an unregistered server has
// nothing else to do, and a master that just dropped the
// stream is usually back within a second, so reconnect
// promptly. Otherwise keep the pulse backoff.
wait = Duration::from_secs(1);
}
tokio::time::sleep(wait).await;
@@ -465,6 +469,9 @@ fn collect_ec_shard_delta_messages(
if quarantined {
continue;
}
// Same numbers the full EC heartbeat carries, read off the active
// generation's index and journal; one call covers every shard.
let (file_count, delete_count) = ec_vol.file_and_delete_count();
for shard in ec_vol.shards.iter().flatten() {
messages.insert(
(
@@ -481,6 +488,8 @@ fn collect_ec_shard_delta_messages(
disk_type: ec_vol.disk_type.to_string(),
expire_at_sec: ec_vol.expire_at_sec,
disk_id: disk_id as u32,
file_count,
delete_count,
encode_ts_ns: ec_vol.encode_ts_ns,
..Default::default()
},
@@ -503,7 +512,17 @@ fn diff_ec_shard_delta_messages(
let mut deleted_ec_shards = Vec::new();
for (key, message) in current {
if previous.get(key) != Some(message) {
// Counts move with every delete; the full EC heartbeat carries those,
// so only a change to the shard itself is news here.
let unchanged = previous.get(key).is_some_and(|prev| {
*message
== master_pb::VolumeEcShardInformationMessage {
file_count: message.file_count,
delete_count: message.delete_count,
..prev.clone()
}
});
if !unchanged {
new_ec_shards.push(message.clone());
}
}
@@ -549,7 +568,7 @@ async fn do_heartbeat(
current_master: &str,
pulse: Duration,
shutdown_rx: &mut broadcast::Receiver<()>,
confirmed_leader_once: &mut bool,
confirmed_leader: &mut bool,
) -> Result<Option<String>, Box<dyn std::error::Error>> {
let channel = connect_channel(
grpc_addr,
@@ -596,7 +615,6 @@ async fn do_heartbeat(
volume_tick.tick().await;
ec_tick.tick().await;
let mut confirmed_leader = false;
loop {
tokio::select! {
resp = response_stream.message() => {
@@ -614,12 +632,11 @@ async fn do_heartbeat(
publish_current_master(state, &hb_resp.leader).await;
return Ok(Some(hb_resp.leader));
}
} else if !confirmed_leader {
} else if !*confirmed_leader {
// The leader answers every new stream without
// naming a leader, so this peer is the one lookups
// go to.
confirmed_leader = true;
*confirmed_leader_once = true;
*confirmed_leader = true;
publish_current_master(state, current_master).await;
}
if !hb_resp.duplicated_uuids.is_empty() {
@@ -1051,15 +1068,17 @@ fn collect_heartbeat_with_snapshot(
master_pb::Heartbeat,
Vec<master_pb::VolumeInformationMessage>,
) {
let (_, expired_ec) = state.store.read().unwrap().find_expired_ec_volumes();
let mut deleted_ec_shards = Vec::new();
let (_, expired_ec, mut deleted_ec_shards) =
state.store.read().unwrap().find_expired_ec_volumes();
if !expired_ec.is_empty() {
deleted_ec_shards = state
.store
.write()
.unwrap()
.remove_expired_ec_volumes(expired_ec)
.0;
deleted_ec_shards.extend(
state
.store
.write()
.unwrap()
.remove_expired_ec_volumes(expired_ec)
.0,
);
}
#[cfg(test)]
{
@@ -2413,6 +2432,169 @@ mod tests {
assert!(volume.is_no_write_or_delete());
}
// A master that is down is redialed after about a second for a pulse after
// the stream to it was lost, since a restarting master is usually back by
// then, and at the pulse pace once the loss is older. Mirrors Go's
// TestHeartbeatOnceKeepsPulseBackoffOnceLossIsOld.
#[tokio::test]
async fn test_heartbeat_once_redials_promptly_only_for_a_pulse_after_a_loss() {
// Closing each connection fails every dial at once, where Windows takes
// about two seconds to refuse a closed port.
let listener = tokio::net::TcpListener::bind("127.0.0.1:0").await.unwrap();
let port = listener.local_addr().unwrap().port();
tokio::spawn(async move {
while let Ok((conn, _)) = listener.accept().await {
drop(conn);
}
});
let down = format!("127.0.0.1:{}.{}", port, port);
let pulse = Duration::from_secs(3);
let mut config = test_config();
config.master_addresses = vec![down.clone()];
config.pulse_seconds = pulse.as_secs();
let temp_dir = tempfile::tempdir().unwrap();
let state = test_state_with_store(reporting_store(temp_dir.path().to_str().unwrap(), 0));
let (_shutdown_tx, mut shutdown_rx) = broadcast::channel(1);
for (lost_ago, prompt) in [(Duration::ZERO, true), (pulse, false)] {
let mut leader_lost_at = Some(tokio::time::Instant::now() - lost_ago);
let mut duplicate_retry_count = 0;
let started = tokio::time::Instant::now();
heartbeat_once(
&config,
&state,
&down,
pulse,
&mut shutdown_rx,
&mut leader_lost_at,
&mut duplicate_retry_count,
)
.await;
let elapsed = started.elapsed();
if prompt {
assert!(
elapsed >= Duration::from_millis(900) && elapsed < Duration::from_secs(2),
"redial right after a loss took {:?}",
elapsed
);
} else {
assert!(
elapsed >= Duration::from_millis(2500),
"redial a pulse after a loss took {:?}",
elapsed
);
}
}
}
/// A master that confirms each heartbeat stream and then ends it, the way
/// one killed right after registering this server looks. It answers
/// nothing else, so no peer names a leader.
#[derive(Clone)]
struct DroppingLeader;
impl tonic::server::NamedService for DroppingLeader {
const NAME: &'static str = "master_pb.Seaweed";
}
impl tonic::codegen::Service<tonic::codegen::http::Request<tonic::body::Body>> for DroppingLeader {
type Response = tonic::codegen::http::Response<tonic::body::Body>;
type Error = std::convert::Infallible;
type Future = tonic::codegen::BoxFuture<Self::Response, Self::Error>;
fn poll_ready(
&mut self,
_cx: &mut std::task::Context<'_>,
) -> std::task::Poll<Result<(), Self::Error>> {
std::task::Poll::Ready(Ok(()))
}
fn call(&mut self, req: tonic::codegen::http::Request<tonic::body::Body>) -> Self::Future {
Box::pin(async move {
if req.uri().path() != "/master_pb.Seaweed/SendHeartbeat" {
return Ok(tonic::Status::unimplemented("").into_http());
}
let mut grpc = tonic::server::Grpc::new(tonic_prost::ProstCodec::default());
Ok(grpc.streaming(ConfirmThenDrop, req).await)
})
}
}
struct ConfirmThenDrop;
impl tonic::server::StreamingService<master_pb::Heartbeat> for ConfirmThenDrop {
type Response = master_pb::HeartbeatResponse;
type ResponseStream = tokio_stream::Iter<
std::vec::IntoIter<Result<master_pb::HeartbeatResponse, tonic::Status>>,
>;
type Future =
tonic::codegen::BoxFuture<tonic::Response<Self::ResponseStream>, tonic::Status>;
fn call(
&mut self,
request: tonic::Request<tonic::Streaming<master_pb::Heartbeat>>,
) -> Self::Future {
Box::pin(async move {
let _ = request.into_inner().message().await;
Ok(tonic::Response::new(tokio_stream::iter(vec![
Ok(master_pb::HeartbeatResponse {
volume_size_limit: 1 << 30,
..Default::default()
}),
Err(tonic::Status::unavailable("master stopped")),
])))
})
}
}
// A server registered long ago whose master just dropped the stream
// redials after about a second, not a pulse. Mirrors Go's
// TestHeartbeatOnceRedialsPromptlyAfterLosingRegisteredStream.
#[tokio::test]
async fn test_heartbeat_once_redials_promptly_after_losing_a_registered_stream() {
let listener = tokio::net::TcpListener::bind("127.0.0.1:0").await.unwrap();
let port = listener.local_addr().unwrap().port();
let master = format!("127.0.0.1:{}.{}", port, port);
tokio::spawn(
tonic::transport::Server::builder()
.add_service(DroppingLeader)
.serve_with_incoming(tokio_stream::wrappers::TcpListenerStream::new(listener)),
);
let pulse = Duration::from_secs(3);
let mut config = test_config();
config.master_addresses = vec![master.clone()];
config.pulse_seconds = pulse.as_secs();
let temp_dir = tempfile::tempdir().unwrap();
let state = test_state_with_store(reporting_store(temp_dir.path().to_str().unwrap(), 0));
let (_shutdown_tx, mut shutdown_rx) = broadcast::channel(1);
let registered_long_ago = tokio::time::Instant::now() - 10 * pulse;
let mut leader_lost_at = Some(registered_long_ago);
let mut duplicate_retry_count = 0;
let started = tokio::time::Instant::now();
heartbeat_once(
&config,
&state,
&master,
pulse,
&mut shutdown_rx,
&mut leader_lost_at,
&mut duplicate_retry_count,
)
.await;
let elapsed = started.elapsed();
assert!(
leader_lost_at.is_some_and(|at| at >= started),
"the end of the confirmed stream was not recorded"
);
assert!(
elapsed >= Duration::from_millis(900) && elapsed < Duration::from_secs(2),
"redial after losing a registered stream took {:?}",
elapsed
);
}
// The pass only reads the store, so it must not shut out the readers that
// serve traffic while it stats every volume.
#[tokio::test]
@@ -2903,6 +3085,46 @@ mod tests {
assert_eq!(last_ec_shards.len(), 1);
}
// The volume heartbeat that quarantines an EC volume tells the master its
// shards are gone, once; neither the next delta nor the next heartbeat
// repeats it.
#[test]
fn test_ec_volume_quarantined_by_a_volume_heartbeat_is_reported_deleted_once() {
let temp_dir = tempfile::tempdir().unwrap();
let dir = temp_dir.path().to_str().unwrap();
let state = test_state_with_store(reporting_store(dir, 0));
mount_test_ec_shard(&state, dir, "ec_delta_kept", 96);
mount_test_ec_shard(&state, dir, "ec_delta_quarantined", 97);
let mut last_ec_shards = collect_ec_shard_delta_messages(&state.store.read().unwrap());
{
let store = state.store.read().unwrap();
let ec_vol = store.find_ec_volume(VolumeId(97)).unwrap();
let eio = crate::storage::volume::media_error_for_test();
for _ in 0..crate::storage::io_error::IO_ERROR_TOLERANCE {
ec_vol.check_read_write_error(Some(&eio));
}
}
let (heartbeat, new_ec_shards, deleted_ec_shards) =
ec_delta_after_volume_tick(&state, &mut last_ec_shards);
assert_eq!(
heartbeat
.deleted_ec_shards
.iter()
.map(|s| s.id)
.collect::<Vec<_>>(),
vec![97]
);
assert!(new_ec_shards.is_empty());
assert!(
deleted_ec_shards.is_empty(),
"a quarantined EC volume was reported deleted twice"
);
let (heartbeat, _, _) = ec_delta_after_volume_tick(&state, &mut last_ec_shards);
assert!(heartbeat.deleted_ec_shards.is_empty());
}
#[test]
fn test_expired_ec_volume_gone_before_removal_is_not_reported_deleted() {
let temp_dir = tempfile::tempdir().unwrap();
@@ -2931,7 +3153,7 @@ mod tests {
.expire_at_sec = 1;
}
let (ec_shards, mut expired) = store.find_expired_ec_volumes();
let (ec_shards, mut expired, _) = store.find_expired_ec_volumes();
expired.sort();
assert!(ec_shards.is_empty());
assert_eq!(expired, vec![(0, VolumeId(71)), (0, VolumeId(72))]);
@@ -3157,4 +3379,88 @@ mod tests {
assert_eq!(deleted_after_delete[0].ec_index_bits, 1);
assert_eq!(deleted_after_delete[0].shard_sizes, vec![0]);
}
#[test]
fn test_diff_ec_shard_delta_messages_ignores_count_changes() {
let key = (81, "ec_delta_case".to_string(), 0, 0);
let previous = HashMap::from([(
key.clone(),
master_pb::VolumeEcShardInformationMessage {
id: 81,
collection: "ec_delta_case".to_string(),
ec_index_bits: 1,
shard_sizes: vec![5],
file_count: 3,
..Default::default()
},
)]);
let mut current = previous.clone();
current.get_mut(&key).unwrap().file_count = 4;
current.get_mut(&key).unwrap().delete_count = 1;
let (new_ec_shards, deleted_ec_shards) = diff_ec_shard_delta_messages(&previous, &current);
assert!(new_ec_shards.is_empty());
assert!(deleted_ec_shards.is_empty());
current.get_mut(&key).unwrap().shard_sizes = vec![6];
let (new_ec_shards, _) = diff_ec_shard_delta_messages(&previous, &current);
assert_eq!(new_ec_shards.len(), 1);
assert_eq!(new_ec_shards[0].delete_count, 1);
}
/// Both EC emission paths report the counts the Go server's
/// `EcVolume.FileAndDeleteCount` defines: file_count is every entry in the
/// active .ecx (live and tombstoned alike), delete_count the needles this
/// holder deleted.
#[test]
fn test_ec_shard_messages_carry_file_and_delete_counts() {
use crate::storage::types::{NEEDLE_MAP_ENTRY_SIZE, Offset, Size, idx_entry_to_bytes};
let temp_dir = tempfile::tempdir().unwrap();
let dir = temp_dir.path().to_str().unwrap();
let mut store = Store::new(NeedleMapKind::InMemory);
store
.add_location(
dir,
dir,
8,
DiskType::HardDrive,
MinFreeSpace::Percent(1.0),
Vec::new(),
)
.unwrap();
std::fs::write(format!("{}/ec_counts_case_91.ec00", dir), b"shard").unwrap();
let mut ecx = Vec::new();
for (key, offset) in [(1u64, 8u32), (5, 200), (10, 500)] {
let mut entry = [0u8; NEEDLE_MAP_ENTRY_SIZE];
idx_entry_to_bytes(
&mut entry,
NeedleId(key),
Offset::from_actual_offset(offset as i64),
Size(100),
);
ecx.extend_from_slice(&entry);
}
std::fs::write(format!("{}/ec_counts_case_91.ecx", dir), &ecx).unwrap();
store.locations[0]
.mount_ec_shards(VolumeId(91), "ec_counts_case", &[0], "")
.unwrap();
let ec_vol = store.locations[0].find_ec_volume_mut(VolumeId(91)).unwrap();
ec_vol.journal_delete(NeedleId(1)).unwrap();
let delta = collect_ec_shard_delta_messages(&store);
let message = delta
.get(&(91, "ec_counts_case".to_string(), 0, 0))
.expect("delta message for the mounted shard");
assert_eq!(message.file_count, 3);
assert_eq!(message.delete_count, 1);
let heartbeat = collect_ec_heartbeat(&test_config(), &store);
assert_eq!(heartbeat.ec_shards.len(), 1);
assert_eq!(heartbeat.ec_shards[0].file_count, 3);
assert_eq!(heartbeat.ec_shards[0].delete_count, 1);
}
}
+626 -26
View File
@@ -40,7 +40,7 @@ use tonic::Request;
use crate::pb::master_pb::{self, LookupEcVolumeRequest};
use crate::pb::volume_server_pb::{
CopyFileRequest, VolumeEcBlobDeleteRequest, VolumeEcShardReadRequest,
CopyFileRequest, VolumeEcBlobDeleteRequest, VolumeEcShardReadRequest, VolumeEcShardReadResponse,
};
use crate::server::grpc_client::{
GrpcDialOptions, connect_channel, master_client, parse_grpc_address, volume_server_client,
@@ -1035,7 +1035,7 @@ fn scrub_snapshot_under_lock(
version: ecv.version,
actual_size: get_actual_size(size, ecv.version) as usize,
size_for_parse: size,
intervals: read_local_intervals(ecv, &intervals),
intervals: read_local_intervals(ecv, &intervals, true),
encode_ts_ns: ecv.encode_ts_ns,
})
}
@@ -1052,20 +1052,29 @@ struct ScrubSnapshot {
}
/// Read any locally-held shard intervals, marking the rest `NeedRemote`. Shared
/// by the read-path `build_snapshot` and the scrub snapshot.
/// by the read-path `build_snapshot` and the scrub snapshot. Shards on a
/// quarantined disk count as not local unless `allow_quarantined` is set.
fn read_local_intervals(
ecv: &crate::storage::erasure_coding::ec_volume::EcVolume,
intervals: &[crate::storage::erasure_coding::ec_locate::Interval],
allow_quarantined: bool,
) -> Vec<IntervalResult> {
let skip_local = !allow_quarantined && ecv.get_io_error_state().2;
let mut interval_results = Vec::with_capacity(intervals.len());
for interval in intervals {
let (shard_id, shard_offset) = ecv.interval_to_shard_id_and_offset(interval);
let buf_size = interval.size as usize;
let local = ecv.shards.get(shard_id as usize).and_then(|s| s.as_ref());
let local = ecv
.shards
.get(shard_id as usize)
.and_then(|s| s.as_ref())
.filter(|_| !skip_local);
match local {
Some(shard) => {
let mut buf = vec![0u8; buf_size];
match shard.read_at(&mut buf, shard_offset as u64) {
let read = shard.read_at(&mut buf, shard_offset as u64);
ecv.check_read_write_error(read.as_ref().err());
match read {
Ok(n) if n == buf_size => interval_results.push(IntervalResult::Local(buf)),
_ => interval_results.push(IntervalResult::NeedRemote {
shard_id,
@@ -1101,7 +1110,7 @@ fn build_snapshot(
));
}
let actual = get_actual_size(size, ecv.version);
let interval_results = read_local_intervals(ecv, intervals);
let interval_results = read_local_intervals(ecv, intervals, false);
let metas = intervals
.iter()
.map(|iv| {
@@ -1356,7 +1365,82 @@ async fn fetch_one_interval(
// Reconstruct: fan-out reads to every other shard at the same
// (shard_offset, size). Mirrors `recoverOneRemoteEcShardInterval`.
recover_one_remote_ec_shard_interval(state, iv, map, &HashSet::from([iv.shard_id])).await
let result =
recover_one_remote_ec_shard_interval(state, iv, map, &HashSet::from([iv.shard_id])).await;
note_ec_reconstructed_interval(iv, &result);
if result.is_err()
&& let Some(buf) = read_quarantined_local_interval(state, iv)
{
tracing::info!(
"read ec shard {}.{} from its quarantined disk: no healthy copy answered",
vid.0,
shard_id
);
return Ok((buf, false));
}
result
}
/// Last resort once peers and reconstruction failed: the interval from a local
/// shard skipped because its disk is quarantined. Mirrors the tail of Go's
/// `readOneEcShardInterval`.
fn read_quarantined_local_interval(
state: &Arc<VolumeServerState>,
iv: EcInterval,
) -> Option<Vec<u8>> {
let store = state.store.read().unwrap();
let ecv = store.find_ec_volume_with_shard(iv.vid, iv.shard_id)?;
if !ecv.get_io_error_state().2
|| (iv.expected_encode_ts_ns != 0 && ecv.encode_ts_ns != iv.expected_encode_ts_ns)
{
return None;
}
let shard = ecv.shards.get(iv.shard_id as usize)?.as_ref()?;
let mut buf = vec![0u8; iv.size];
let read = shard.read_at(&mut buf, iv.shard_offset as u64);
ecv.check_read_write_error(read.as_ref().err());
match read {
Ok(n) if n == iv.size => Some(buf),
result => {
tracing::info!(
"read quarantined ec shard {}.{} offset {}: {:?}",
iv.vid.0,
iv.shard_id,
iv.shard_offset,
result
);
None
}
}
}
/// Counts one interval that couldn't be read directly and was reconstructed,
/// and at debug logs it with the needle and the encode generation, which the
/// counter can't carry. `success` means the reconstruction produced data; the
/// needle is validated later. A holder's "deleted" answer with too few shards
/// rebuilt nothing, so it is a failure, as Go's reconstruct error makes it there.
fn note_ec_reconstructed_interval(iv: EcInterval, result: &io::Result<(Vec<u8>, bool)>) {
let outcome = match result {
Ok((buf, _)) if !buf.is_empty() => "success",
_ => "failure",
};
crate::metrics::EC_RECONSTRUCTED_INTERVAL_COUNTER
.with_label_values(&[outcome])
.inc();
// tracing evaluates these arguments only when debug is enabled.
tracing::debug!(
"ec reconstructed interval: volume={} generation={} needle={} shard={} result={} error={}",
iv.vid.0,
iv.expected_encode_ts_ns,
iv.needle_id,
iv.shard_id,
outcome,
match result {
Ok((buf, _)) if !buf.is_empty() => "<nil>".to_string(),
Ok(_) => "a holder reported the needle deleted".to_string(),
Err(e) => e.to_string(),
}
);
}
async fn read_remote_ec_shard_interval(
@@ -1429,15 +1513,30 @@ async fn do_read_remote_ec_shard_interval(
vid.0, shard_id, source, e
))
})?;
let mut stream = resp.into_inner();
collect_ec_shard_read(resp.into_inner(), source, iv).await
}
/// Reads one `VolumeEcShardRead` reply into the interval's bytes, or the
/// holder's deleted answer, which stands even if the stream fails after it.
async fn collect_ec_shard_read(
mut stream: tonic::Streaming<VolumeEcShardReadResponse>,
source: &str,
iv: EcInterval,
) -> io::Result<(Vec<u8>, bool)> {
let EcInterval {
vid,
shard_id,
size,
expected_encode_ts_ns,
..
} = iv;
let mut out = Vec::with_capacity(size);
let mut is_deleted = false;
while let Some(msg) = stream
.message()
.await
.map_err(|e| io::Error::other(format!("recv: {}", e)))?
{
// A deleted answer stands even if the stream fails after it.
while let Some(msg) = match stream.message().await {
Err(_) if is_deleted => None,
msg => msg.map_err(|e| io::Error::other(format!("recv: {}", e)))?,
} {
// Validate the served shard's identity client-side, so the guard holds even
// against a pre-upgrade server that ignored the request field (returns 0).
// A mismatch fails the read; the caller recovers from parity.
@@ -1516,6 +1615,7 @@ async fn gather_ec_shard_intervals(
// shard_locations cache or a failed master lookup, local
// survivors still contribute.
let mut available = 0usize;
let mut quarantined = Vec::new();
{
let store = state.store.read().unwrap();
for (sid, slot) in bufs.iter_mut().enumerate() {
@@ -1541,13 +1641,15 @@ async fn gather_ec_shard_intervals(
}
_ => continue,
};
if owner.get_io_error_state().2 {
quarantined.push(sid_shard);
continue;
}
if let Some(Some(shard)) = owner.shards.get(sid) {
let mut buf = vec![0u8; size];
if shard
.read_at(&mut buf, shard_offset as u64)
.map(|n| n == size)
.unwrap_or(false)
{
let read = shard.read_at(&mut buf, shard_offset as u64);
owner.check_read_write_error(read.as_ref().err());
if read.map(|n| n == size).unwrap_or(false) {
*slot = Some(buf);
available += 1;
}
@@ -1623,6 +1725,31 @@ async fn gather_ec_shard_intervals(
}
}
// Healthy sources fell short: fill in from the quarantined local disk.
for sid in quarantined {
if any_deleted || available >= want {
break;
}
if bufs[sid as usize].is_some() {
continue;
}
if let Some(buf) = read_quarantined_local_interval(
state,
EcInterval {
shard_id: sid,
..iv
},
) {
tracing::debug!(
"recover: read ec shard {}.{} from its quarantined disk",
vid.0,
sid
);
bufs[sid as usize] = Some(buf);
available += 1;
}
}
(bufs, any_deleted)
}
@@ -1767,10 +1894,11 @@ async fn try_ec_shard_needle_candidate(
/// Repair a needle whose reassembled bytes failed to parse — Go's
/// `rebuildEcShardNeedle`. Every interval's source shard is a suspect: rebuild
/// all suspect ranges at once while parity covers the suspect set, or one
/// suspect at a time when the needle spans more shards than that, and let a
/// successful parse arbitrate which copy was corrupt. A holder's deleted report
/// stops the search: the needle is gone whatever the local index says.
/// all suspect ranges at once while parity covers the suspect set, then search
/// the sets of shards that may be corrupt, then rebuild one suspect at a time,
/// and let a successful parse arbitrate which copy was corrupt. A holder's
/// deleted report stops the search: the needle is gone whatever the local index
/// says.
async fn rebuild_ec_shard_needle(
state: &Arc<VolumeServerState>,
metas: &[EcInterval],
@@ -1808,9 +1936,19 @@ async fn rebuild_ec_shard_needle(
return (Some(candidate), false);
}
}
if suspects.len() == 1 {
return (None, false);
}
}
let (candidate, is_deleted) = rebuild_ec_needle_from_corrupt_sets(
state, metas, map, corrupted, &suspects, offset, size, version, needle_id,
)
.await;
if is_deleted {
return (None, true);
}
if let Some(candidate) = candidate {
return (Some(candidate), false);
}
if suspects.len() == 1 {
return (None, false);
}
for shard_id in suspects {
let (candidate, is_deleted) = try_ec_shard_needle_candidate(
@@ -1954,6 +2092,239 @@ async fn rebuild_ec_needle_from_dropped_inputs(
(None, false)
}
/// Bounds on one needle's corrupt-set search, as Go's `ecRebuildMaxSets` and
/// `ecRebuildMaxBytes`: sets enumerated, and needle bytes rebuilt and parsed
/// past the single-shard sets. 10+4 has 1470 sets of at most four shards.
const EC_REBUILD_MAX_SETS: usize = 4096;
const EC_REBUILD_MAX_BYTES: usize = 1 << 30;
/// Handle a rebuild the retries above could not repair: more shards than one
/// can be corrupt over the same ranges and get read as reconstruction inputs.
/// Each interval is gathered once from every other shard that answers, then
/// `search_ec_corrupt_sets` tries the candidate corrupt sets. `None` when that
/// gather would outgrow the recovery budget. Mirrors Go's
/// `rebuildEcNeedleFromCorruptSets`.
async fn rebuild_ec_needle_from_corrupt_sets(
state: &Arc<VolumeServerState>,
metas: &[EcInterval],
map: EcShardMap<'_>,
corrupted: &[u8],
suspects: &HashSet<ShardId>,
offset: i64,
size: Size,
version: Version,
needle_id: NeedleId,
) -> (Option<Vec<u8>>, bool) {
let EcShardMap {
data_shards,
parity_shards,
..
} = map;
let total_shards = data_shards + parity_shards;
let total_size: usize = metas.iter().map(|iv| iv.size).sum();
if total_size * (total_shards - 1) > EC_RECOVER_BUDGET {
return (None, false);
}
let Ok(permit) = EC_RECOVER_SEM
.acquire_many((total_size * total_shards).min(EC_RECOVER_BUDGET) as u32)
.await
else {
return (None, false);
};
let mut gathers: Vec<Vec<Option<Vec<u8>>>> = Vec::with_capacity(metas.len());
for iv in metas {
let (bufs, is_deleted) =
gather_ec_shard_intervals(state, *iv, map, &HashSet::from([iv.shard_id]), total_shards)
.await;
if is_deleted {
return (None, true);
}
gathers.push(bufs);
}
// The search can run hundreds of decodes, so keep it off the runtime. The
// permit goes with it: a dropped read does not stop a blocking task.
let (metas_owned, corrupted, suspects) = (metas.to_vec(), corrupted.to_vec(), suspects.clone());
let search = tokio::task::spawn_blocking(move || {
let _permit = permit;
let rs = ReedSolomon::new(data_shards, parity_shards).ok()?;
search_ec_corrupt_sets(
&mut gathers,
&metas_owned,
&rs,
&corrupted,
&suspects,
|candidate| {
let mut check = Needle {
id: needle_id,
..Needle::default()
};
check.read_bytes(candidate, offset, size, version).is_ok() && check.id == needle_id
},
)
});
let Ok(Some((candidate, convicted))) = search.await else {
return (None, false);
};
if let Some(first) = metas.first() {
for shard_id in convicted {
forget_shard_location(state, first.vid, shard_id);
}
}
(Some(candidate), false)
}
/// Try each set of up to `parity_shards` shards holding a suspect as the
/// corrupt set, smallest first: suspect ranges inside it are rebuilt from the
/// shards outside it, the rest keep the bytes read, and `parses` arbitrates. A
/// set whose decodes read the same inputs as one already tried is skipped.
/// Returns the candidate with the shards it convicts: every differing range,
/// and every other shard the set kept out of the decodes.
fn search_ec_corrupt_sets(
gathers: &mut [Vec<Option<Vec<u8>>>],
metas: &[EcInterval],
rs: &ReedSolomon,
corrupted: &[u8],
suspects: &HashSet<ShardId>,
parses: impl Fn(&[u8]) -> bool,
) -> Option<(Vec<u8>, Vec<ShardId>)> {
let data_shards = rs.data_shard_count();
let total_shards = rs.total_shard_count();
let mut candidate = corrupted.to_vec();
let mut scratch: Vec<Vec<u8>> = vec![Vec::new(); data_shards];
let mut inputs = vec![0u64; metas.len()];
let mut tried: HashSet<Vec<u64>> = HashSet::new();
// Records in `inputs` the shards each interval's decode reads when `set`
// is excluded, 0 for an interval that keeps its bytes. False when an
// interval has too few shards left, or the same decodes were tried before.
let mut plan = |set: u64, inputs: &mut [u64], gathers: &[Vec<Option<Vec<u8>>>]| {
for (i, iv) in metas.iter().enumerate() {
inputs[i] = 0;
if set & (1 << iv.shard_id) == 0 {
continue;
}
let mut taken = 0;
for (sid, slot) in gathers[i].iter().enumerate() {
if taken == data_shards {
break;
}
if set & (1 << sid) == 0 && slot.is_some() {
inputs[i] |= 1 << sid;
taken += 1;
}
}
if taken < data_shards {
return false;
}
}
tried.insert(inputs.to_vec())
};
// Rebuilds the planned intervals into `candidate`; reset on failure.
let mut decode =
|inputs: &[u64], candidate: &mut [u8], gathers: &mut [Vec<Option<Vec<u8>>>]| {
let mut pos = 0usize;
for (i, iv) in metas.iter().enumerate() {
let end = pos + iv.size;
if inputs[i] != 0 {
let Some(buf) =
decode_ec_interval(rs, &mut gathers[i], inputs[i], &mut scratch, iv)
else {
candidate.copy_from_slice(corrupted);
return false;
};
candidate[pos..end].copy_from_slice(buf);
}
pos = end;
}
if parses(candidate) {
return true;
}
candidate.copy_from_slice(corrupted);
false
};
let (mut sets, mut spent) = (0usize, 0usize);
for k in 1..=rs.parity_shard_count().min(total_shards) {
let mut members: Vec<usize> = (0..k).collect();
loop {
let set = members.iter().fold(0u64, |set, &sid| set | 1 << sid);
let holds_suspect = members
.iter()
.any(|&sid| suspects.contains(&(sid as ShardId)));
if holds_suspect {
sets += 1;
if sets > EC_REBUILD_MAX_SETS {
return None;
}
if plan(set, &mut inputs, gathers) {
if k > 1 {
if spent >= EC_REBUILD_MAX_BYTES {
return None;
}
spent += corrupted.len();
}
if decode(&inputs, &mut candidate, gathers) {
let mut convicted: Vec<ShardId> = Vec::new();
let mut pos = 0usize;
for iv in metas {
let end = pos + iv.size;
if candidate[pos..end] != corrupted[pos..end] {
convicted.push(iv.shard_id);
}
pos = end;
}
convicted.extend(
members
.iter()
.map(|&sid| sid as ShardId)
.filter(|sid| !suspects.contains(sid)),
);
return Some((candidate, convicted));
}
}
}
let Some(j) = (0..k).rev().find(|&j| members[j] != total_shards - k + j) else {
break;
};
members[j] += 1;
for m in j + 1..k {
members[m] = members[m - 1] + 1;
}
}
}
None
}
/// Decode `iv`'s data shard from the gathered shards in `inputs`, using
/// `scratch`, one buffer per data shard, for every data shard it lacks.
fn decode_ec_interval<'a>(
rs: &ReedSolomon,
slots: &mut [Option<Vec<u8>>],
inputs: u64,
scratch: &'a mut [Vec<u8>],
iv: &EcInterval,
) -> Option<&'a [u8]> {
let mut spare = scratch.iter_mut();
let mut view: Vec<(&mut [u8], bool)> = slots
.iter_mut()
.enumerate()
.map(|(sid, slot)| match (slot, spare.next()) {
(Some(shard), _) if inputs & (1 << sid) != 0 => (shard.as_mut_slice(), true),
(_, Some(buf)) => {
buf.resize(iv.size, 0);
(buf.as_mut_slice(), false)
}
_ => (&mut [][..], false),
})
.collect();
rs.reconstruct_data(&mut view).ok()?;
drop(view);
Some(&scratch[iv.shard_id as usize])
}
// parse_grpc_address lives in `grpc_client.rs` and is re-exported
// here via the use above so this module shares a single
// HTTP↔gRPC port-translation routine with grpc_server.rs.
@@ -2162,7 +2533,7 @@ async fn fetch_ec_index_from_one_peer(
match client.copy_file(copy_req(".vif", true)).await {
Ok(resp) => {
if let Err(e) = drain_copy_stream(resp.into_inner(), vif_path).await {
if let Err(e) = drain_optional_copy_stream(resp.into_inner(), vif_path).await {
tracing::warn!(volume_id = m.vid.0, peer = %peer, "copy .vif: {}", e);
}
}
@@ -2172,6 +2543,48 @@ async fn fetch_ec_index_from_one_peer(
Ok(())
}
/// Drain an optional CopyFile stream into `<dest_path>.copying` and rename it
/// into place only once the source sent bytes, so a peer without the file or a
/// failed stream leaves an existing destination untouched, as Go's writeToFile
/// does for ignore_source_file_not_found.
pub(crate) async fn drain_optional_copy_stream(
stream: tonic::Streaming<crate::pb::volume_server_pb::CopyFileResponse>,
dest_path: &str,
) -> io::Result<()> {
let staging = format!("{}.copying", dest_path);
let mut incomplete = RemoveIncomplete::new(&staging);
let drained = drain_copy_stream(stream, &staging).await;
let has_bytes = fs::metadata(&staging).is_ok_and(|meta| meta.len() > 0);
if drained.is_ok() && has_bytes {
fs::rename(&staging, dest_path)
.map_err(|e| io::Error::other(format!("commit {}: {}", dest_path, e)))?;
incomplete.committed();
}
drained
}
/// Removes a staged copy unless it was committed, including when the copy is
/// dropped mid-stream because its RPC was cancelled.
pub(crate) struct RemoveIncomplete(Option<String>);
impl RemoveIncomplete {
pub(crate) fn new(staging: &str) -> Self {
Self(Some(staging.to_string()))
}
pub(crate) fn committed(&mut self) {
self.0 = None;
}
}
impl Drop for RemoveIncomplete {
fn drop(&mut self) {
if let Some(staging) = self.0.take() {
let _ = fs::remove_file(staging);
}
}
}
/// Drain a CopyFile stream into a local file, truncating it first.
async fn drain_copy_stream(
mut stream: tonic::Streaming<crate::pb::volume_server_pb::CopyFileResponse>,
@@ -2210,6 +2623,139 @@ mod tests {
);
}
const BLOCK: usize = 4096;
/// One row of 10+4 shards holding a needle at `dat_offset`, the needle's
/// intervals, and its bytes.
fn ec_row_with_needle(
dat_offset: usize,
data_len: usize,
) -> (Vec<Vec<u8>>, Vec<EcInterval>, Vec<u8>, Needle) {
let mut n = Needle {
id: NeedleId(7),
data: (0..data_len).map(|i| (i * 31 + 7) as u8).collect(),
..Needle::default()
};
let bytes = n.write_bytes(VERSION_3);
let mut dat = vec![0u8; 10 * BLOCK];
dat[dat_offset..dat_offset + bytes.len()].copy_from_slice(&bytes);
let mut shards: Vec<Vec<u8>> = dat.chunks(BLOCK).map(|c| c.to_vec()).collect();
shards.resize(14, vec![0u8; BLOCK]);
ReedSolomon::new(10, 4)
.unwrap()
.encode(&mut shards)
.unwrap();
let mut metas = Vec::new();
let (mut pos, end) = (dat_offset, dat_offset + bytes.len());
while pos < end {
let size = (BLOCK - pos % BLOCK).min(end - pos);
metas.push(EcInterval {
vid: VolumeId(1),
needle_id: n.id,
shard_id: (pos / BLOCK) as ShardId,
shard_offset: (pos % BLOCK) as i64,
size,
expected_encode_ts_ns: 0,
});
pos += size;
}
(shards, metas, bytes, n)
}
fn corrupt(shards: &mut [Vec<u8>], shard_ids: &[ShardId], metas: &[EcInterval]) {
for &sid in shard_ids {
for iv in metas {
let start = iv.shard_offset as usize;
for b in &mut shards[sid as usize][start..start + iv.size] {
*b ^= 0xFF;
}
}
}
}
fn search(
shards: &[Vec<u8>],
metas: &[EcInterval],
n: &Needle,
dat_offset: usize,
) -> Option<(Vec<u8>, Vec<ShardId>)> {
let mut corrupted = Vec::new();
let mut gathers = Vec::new();
for iv in metas {
let range = iv.shard_offset as usize..iv.shard_offset as usize + iv.size;
corrupted.extend_from_slice(&shards[iv.shard_id as usize][range.clone()]);
gathers.push(
shards
.iter()
.enumerate()
.map(|(sid, shard)| {
(sid != iv.shard_id as usize).then(|| shard[range.clone()].to_vec())
})
.collect(),
);
}
let suspects: HashSet<ShardId> = metas.iter().map(|iv| iv.shard_id).collect();
let rs = ReedSolomon::new(10, 4).unwrap();
search_ec_corrupt_sets(
&mut gathers,
metas,
&rs,
&corrupted,
&suspects,
|candidate| {
let mut check = Needle {
id: n.id,
..Needle::default()
};
check
.read_bytes(candidate, dat_offset as i64, n.size, VERSION_3)
.is_ok()
&& check.id == n.id
},
)
}
#[test]
fn search_ec_corrupt_sets_rebuilds_with_parity_many_shards_corrupt() {
let dat_offset = 2 * BLOCK + 100;
let (mut shards, metas, bytes, n) = ec_row_with_needle(dat_offset, 1000);
assert_eq!(metas.len(), 1);
// The needle's own shard and the lowest-numbered survivors, which every
// first decode reads.
corrupt(&mut shards, &[2, 0, 1, 3], &metas);
let (rebuilt, mut convicted) = search(&shards, &metas, &n, dat_offset).unwrap();
assert_eq!(rebuilt, bytes);
convicted.sort();
assert_eq!(convicted, vec![0, 1, 2, 3]);
}
#[test]
fn search_ec_corrupt_sets_keeps_a_clean_suspect_as_input() {
let dat_offset = BLOCK - 300;
let (mut shards, metas, bytes, n) = ec_row_with_needle(dat_offset, 1000);
assert_eq!(metas.len(), 2);
// Shard 0 is corrupt and shard 1 clean; rebuilding both would exceed
// parity with three other shards corrupt too.
corrupt(&mut shards, &[0], &metas[..1]);
corrupt(&mut shards, &[2, 3, 4], &metas);
let (rebuilt, mut convicted) = search(&shards, &metas, &n, dat_offset).unwrap();
assert_eq!(rebuilt, bytes);
convicted.sort();
assert_eq!(convicted, vec![0, 2, 3, 4]);
}
#[test]
fn search_ec_corrupt_sets_fails_past_parity() {
let dat_offset = 100;
let (mut shards, metas, _, n) = ec_row_with_needle(dat_offset, 1000);
corrupt(&mut shards, &[0, 1, 2, 3, 4], &metas);
assert!(search(&shards, &metas, &n, dat_offset).is_none());
}
fn locations(count: usize) -> HashMap<ShardId, Vec<String>> {
(0..count)
.map(|sid| (sid as ShardId, vec!["127.0.0.1:8080".to_string()]))
@@ -2235,4 +2781,58 @@ mod tests {
// An unrefreshed cache always looks up.
assert!(needs_refresh(&locations(0), None, false, 10, 14));
}
struct FailAfterFrame(Option<bytes::Bytes>);
impl http_body::Body for FailAfterFrame {
type Data = bytes::Bytes;
type Error = tonic::Status;
fn poll_frame(
mut self: std::pin::Pin<&mut Self>,
_cx: &mut std::task::Context<'_>,
) -> std::task::Poll<Option<Result<http_body::Frame<bytes::Bytes>, tonic::Status>>>
{
std::task::Poll::Ready(Some(match self.0.take() {
Some(frame) => Ok(http_body::Frame::data(frame)),
None => Err(tonic::Status::unavailable(
"stream failed after the deleted answer",
)),
}))
}
}
/// A holder's deleted answer stands even when its stream fails afterwards,
/// so a later source or the quarantined shard cannot answer for the needle.
#[tokio::test]
async fn test_collect_ec_shard_read_keeps_deletion_when_the_stream_fails() {
let message = prost::Message::encode_to_vec(&VolumeEcShardReadResponse {
is_deleted: true,
..Default::default()
});
let mut frame = vec![0u8];
frame.extend_from_slice(&(message.len() as u32).to_be_bytes());
frame.extend_from_slice(&message);
let stream = tonic::Streaming::new_response(
tonic_prost::ProstDecoder::<VolumeEcShardReadResponse>::default(),
FailAfterFrame(Some(frame.into())),
tonic::codegen::http::StatusCode::OK,
None,
None,
);
let iv = EcInterval {
vid: VolumeId(7),
needle_id: NeedleId(42),
shard_id: 0,
shard_offset: 0,
size: 64,
expected_encode_ts_ns: 0,
};
let (buf, is_deleted) = collect_ec_shard_read(stream, "peer:18080", iv)
.await
.unwrap();
assert!(is_deleted);
assert!(buf.is_empty());
}
}
+42 -14
View File
@@ -18,7 +18,7 @@ use tracing::debug;
use crate::storage::needle::needle::Needle;
use crate::storage::types::{Size, VolumeId};
use crate::storage::volume::VolumeError;
use crate::storage::volume::{PendingWrite, VolumeError};
use super::volume_server::VolumeServerState;
@@ -26,7 +26,12 @@ use super::volume_server::VolumeServerState;
pub type WriteResult = Result<(u64, Size, bool), VolumeError>;
/// The needles queued for one volume, each with the durability it asked for.
type VolumeBatch = Vec<(Needle, bool, oneshot::Sender<WriteResult>)>;
type VolumeBatch = Vec<(
Needle,
bool,
oneshot::Sender<WriteResult>,
Option<PendingWrite>,
)>;
/// A request to write a needle, submitted to the write queue.
pub struct WriteRequest {
@@ -38,6 +43,8 @@ pub struct WriteRequest {
/// that cancels `submit` after the send must not free the slot while the
/// worker still holds the needle.
pub queue_permit: Option<OwnedSemaphorePermit>,
/// Tracked from submission, so a delete cancels the write while it waits.
pub pending: Option<PendingWrite>,
}
/// Maximum number of write requests to batch together.
@@ -167,12 +174,13 @@ impl WriteQueue {
#[cfg(test)]
self.submits
.fetch_add(1, std::sync::atomic::Ordering::Relaxed);
{
let pending = {
let store = self.state.store.read().unwrap();
if store.find_volume(volume_id).is_none() {
return Err(VolumeError::NotFound);
match store.find_volume(volume_id) {
Some((_, vol)) => vol.track_pending_write(&needle),
None => return Err(VolumeError::NotFound),
}
}
};
// The permit travels inside the request, so the global queued bound
// covers the request until the worker drops it — including after a
// canceled submit.
@@ -189,6 +197,7 @@ impl WriteQueue {
fsync,
response_tx,
queue_permit: Some(queue_permit),
pending: Some(pending),
};
// A send fails only against a retiring worker's closed channel; the
@@ -345,10 +354,11 @@ fn process_batch(state: Arc<VolumeServerState>, batch: Vec<WriteRequest>) {
for req in batch {
let vid = req.volume_id;
let entry = (req.needle, req.fsync, req.response_tx, req.pending);
if let Some(group) = groups.iter_mut().find(|(v, _)| *v == vid) {
group.1.push((req.needle, req.fsync, req.response_tx));
group.1.push(entry);
} else {
groups.push((vid, vec![(req.needle, req.fsync, req.response_tx)]));
groups.push((vid, vec![entry]));
}
}
@@ -356,16 +366,29 @@ fn process_batch(state: Arc<VolumeServerState>, batch: Vec<WriteRequest>) {
// the group's I/O with the guard released — one volume's write must not
// block admission on another volume's.
for (vid, entries) in groups {
let (mut writes, senders): (Vec<_>, Vec<_>) = entries
.into_iter()
.map(|(needle, fsync, response_tx)| ((needle, fsync), response_tx))
.unzip();
let vol = {
let store = state.store.read().unwrap();
store.writable_volume(vid)
};
let results = match vol {
Ok(vol) => vol.write_needles_grouped(&mut writes),
let mut writes = Vec::with_capacity(entries.len());
let mut senders = Vec::with_capacity(entries.len());
let mut pendings = Vec::with_capacity(entries.len());
for (needle, fsync, response_tx, pending) in entries {
// A delete reaching a remounted volume cannot cancel a write
// queued for the old one.
if let (Ok(vol), Some(tracked)) = (&vol, &pending)
&& !vol.tracks_pending_write(tracked)
{
drop(pending);
let _ = response_tx.send(Err(VolumeError::NotFound));
continue;
}
writes.push((needle, fsync));
senders.push(response_tx);
pendings.push(pending);
}
let results: Vec<WriteResult> = match vol {
Ok(vol) => vol.write_pending_needles(&mut writes, &pendings),
Err(e) => writes
.iter()
.map(|_| match e {
@@ -374,6 +397,9 @@ fn process_batch(state: Arc<VolumeServerState>, batch: Vec<WriteRequest>) {
})
.collect(),
};
// Untrack before replying, as Go does: a caller that sees its write
// finish must not still find it pending.
drop(pendings);
for (response_tx, result) in senders.into_iter().zip(results) {
// Send result back; ignore error if receiver dropped.
let _ = response_tx.send(result);
@@ -577,6 +603,7 @@ mod tests {
fsync: true,
response_tx,
queue_permit: None,
pending: None,
}
})
.collect();
@@ -669,6 +696,7 @@ mod tests {
fsync: false,
response_tx,
queue_permit: None,
pending: None,
}];
let writer = {
let state = state.clone();
@@ -581,6 +581,51 @@ pub fn verify_shard_blocks(
Ok(mismatched)
}
/// Checks the data shards a decode reads against the volume's bitrot
/// protection: a corrupt shard copied into the .dat is lost for good once the
/// caller deletes the parity that could repair it. Mirrors Go's
/// `VerifyEcVolumeDataShards`.
pub fn verify_ec_volume_data_shards(
prot: Option<&EcBitrotProtection>,
status: BitrotStatus,
shard_paths: &[String],
) -> io::Result<()> {
let prot = match (status, prot) {
(BitrotStatus::Off, _) => return Ok(()),
(BitrotStatus::On, Some(prot)) => prot,
_ => {
return Err(io::Error::new(
io::ErrorKind::InvalidData,
"bitrot sidecar is malformed or unverifiable",
));
}
};
for (shard_id, path) in shard_paths.iter().enumerate() {
let Some(entry) = shard_checksums(prot, shard_id as u32) else {
continue;
};
let mismatched =
verify_shard_file_blocks(path, entry, prot.block_size as i64).map_err(|e| {
io::Error::new(
e.kind(),
format!("verify data shard {} {}: {}", shard_id, path, e),
)
})?;
if !mismatched.is_empty() {
return Err(io::Error::new(
io::ErrorKind::InvalidData,
format!(
"data shard {} {} fails its bitrot checksum in {} blocks",
shard_id,
path,
mismatched.len()
),
));
}
}
Ok(())
}
/// Builds the `EcShardConfig` proto for the given layout. The bitrot sidecar
/// carries its own top-level encode_uuid, so the nested config leaves it empty.
pub fn ec_shard_config(data_shards: u32, parity_shards: u32, block_size: i64) -> EcShardConfig {
@@ -238,8 +238,8 @@ pub struct EcVolume {
/// no encode identity, so provenance is the only signal that a loaded
/// manifest may describe a different encode run. Same purpose as
/// `ecx_actual_dir`. Empty when no sidecar was found. Always committed in
/// the same step as `bitrot`/`bitrot_status`, so a reload whose `Err` gets
/// swallowed by `reload_bitrot_sidecar` cannot leave this describing a
/// the same step as `bitrot`/`bitrot_status`, so a reload whose `Err` a
/// caller ignores cannot leave this describing a
/// sidecar other than the one those two fields actually hold.
pub(crate) bitrot_source_dir: String,
@@ -802,14 +802,17 @@ impl EcVolume {
/// shard delivery can bring the manifest with it, and the receive path only
/// writes the file, so without this the in-memory volume keeps the
/// protection state it resolved at mount (off) until a remount.
pub fn reload_bitrot_sidecar(&mut self, additional_dirs: &[String]) {
if let Err(e) = self.load_active_bitrot_sidecar(additional_dirs) {
/// Returns the layout-mismatch error a mount would fail on.
pub fn reload_bitrot_sidecar(&mut self, additional_dirs: &[String]) -> io::Result<()> {
let result = self.load_active_bitrot_sidecar(additional_dirs);
if let Err(e) = &result {
tracing::warn!(
volume_id = self.volume_id.0,
error = %e,
"reload bitrot sidecar",
);
}
result
}
/// Load the generation-0 checksum sidecar into `self.bitrot`/`self.bitrot_status`.
@@ -888,8 +891,8 @@ impl EcVolume {
self.parity_shards as usize,
);
// Provenance is committed together with the sidecar it describes, so a
// swallowed reload (`reload_bitrot_sidecar` logs and discards the `Err`
// above) cannot leave them disagreeing: the geometry-mismatch return
// reload whose `Err` above a caller ignores cannot leave them
// disagreeing: the geometry-mismatch return
// happens before this point, so `bitrot`/`bitrot_status` below and
// `bitrot_source_dir` here always describe the same load attempt.
// Only a path that actually exists is a real source: `path` falls back
@@ -934,6 +937,31 @@ impl EcVolume {
(self.bitrot.clone(), self.bitrot_status)
}
/// The protection of a volume's per-disk runtimes: the first `On`, else
/// `Invalid`, else `Off`. After a restart only the runtime whose disk holds
/// the sidecar resolves it. Mirrors Go's `SelectBitrotProtection`.
pub(crate) fn select_bitrot_protection(
volumes: &[&EcVolume],
) -> (
Option<crate::pb::volume_server_pb::EcBitrotProtection>,
crate::storage::erasure_coding::ec_bitrot::BitrotStatus,
) {
use crate::storage::erasure_coding::ec_bitrot::BitrotStatus;
let mut invalid = false;
for v in volumes {
match v.bitrot_protection() {
(prot, BitrotStatus::On) => return (prot, BitrotStatus::On),
(_, BitrotStatus::Invalid) => invalid = true,
_ => {}
}
}
if invalid {
(None, BitrotStatus::Invalid)
} else {
(None, BitrotStatus::Off)
}
}
/// Read-only EC bitrot checksum scrub of the LOCAL shards of this volume's
/// active generation.
///
@@ -165,7 +165,8 @@ impl SortedFileNeedleMap {
return Ok(None);
}
let file = pooled_index_files().borrow(&self.db_file_name, true)?;
let Some((entry_index, _, size)) = search_sorted_index(&file, self.db_file_size, key)?
let Some((entry_index, needle_offset, size)) =
search_sorted_index(&file, self.db_file_size, key)?
else {
return Ok(None);
};
@@ -182,22 +183,14 @@ impl SortedFileNeedleMap {
// already counts as deleted.
self.append_to_index_file(key, offset, TOMBSTONE_FILE_SIZE)?;
// Move the counters to where a reload would put them, so the heartbeat
// does not report this volume as garbage-free until it restarts. The
// tombstone is one more .idx row, and under the rule index_metric
// applies both it and the row it supersedes count as deletions; the
// bytes leave the live set without leaving the .idx.
//
// This belongs to the `.idx` append, not to the `.sdx` mark below: the
// append is what makes the delete durable and what a reload would
// count, and the retry path returns early once the overlay records it.
// Leaving the counters until after the mark would strand them at their
// pre-delete values whenever the mark fails.
self.metric.file_count.fetch_add(1, Ordering::Relaxed);
self.metric.deletion_count.fetch_add(2, Ordering::Relaxed);
self.metric
.deletion_byte_count
.fetch_add(size.0.max(0) as u64, Ordering::Relaxed);
// Count against the `.idx` append, not the `.sdx` mark below: the
// append is what a reload counts, and the retry path returns early
// once the overlay records it, so a failed mark must not strand the
// counters at their pre-delete values.
self.metric.on_delete(&NeedleValue {
offset: needle_offset,
size,
});
// From here the delete is durable in `.idx`, so no lookup may report
// this needle live again. Record it before touching `.sdx` and clear
@@ -449,11 +442,11 @@ fn write_sorted_file_from_idx(
/// its bloom filter — and so without its false positives or its per-volume
/// allocation.
///
/// Go counts every `.idx` row in `FileCounter`, and counts a row in
/// `DeletionCounter` when it is not the last, still-valid row for its key. The
/// rows that *are* the last valid row for their key are exactly the entries
/// `.sdx` holds, so `DeletionCounter = rows - live` and
/// `DeletionByteCounter = valid bytes in .idx - live bytes in .sdx`, each
/// Go counts every `.idx` put row in `FileCounter` and tombstones in neither
/// counter, so a key costs one deletion per put that is no longer its live
/// row. The live rows are exactly the unmarked entries `.sdx` holds, so
/// `DeletionCounter = puts - live` and
/// `DeletionByteCounter = put bytes in .idx - live bytes in .sdx`, each
/// computable in one sequential pass with no per-key state.
fn index_metric(
index_file_name: &str,
@@ -462,15 +455,17 @@ fn index_metric(
) -> io::Result<NeedleMapMetric> {
let metric = NeedleMapMetric::default();
let mut rows: i64 = 0;
let mut valid_bytes: u64 = 0;
let is_put = |offset: Offset, size: Size| !offset.is_zero() && !size.is_deleted();
let mut puts: i64 = 0;
let mut put_bytes: u64 = 0;
let mut index_file = File::open(index_file_name)?;
idx::walk_index_file(&mut index_file, 0, |key, offset, size| {
rows += 1;
metric.maybe_set_max_file_key(key);
metric.maybe_set_max_needle_end(offset, size, version);
if size.is_valid() {
valid_bytes += size.0 as u64;
if is_put(offset, size) {
puts += 1;
put_bytes += size.0 as u64;
}
Ok(())
})?;
@@ -478,22 +473,22 @@ fn index_metric(
let mut live: i64 = 0;
let mut live_bytes: u64 = 0;
let mut db_file = File::open(db_file_name)?;
idx::walk_index_file(&mut db_file, 0, |_, _, size| {
if size.is_valid() {
idx::walk_index_file(&mut db_file, 0, |_, offset, size| {
if is_put(offset, size) {
live += 1;
live_bytes += size.0 as u64;
}
Ok(())
})?;
metric.file_count.store(rows, Ordering::Relaxed);
metric.file_byte_count.store(valid_bytes, Ordering::Relaxed);
metric.file_count.store(puts, Ordering::Relaxed);
metric.file_byte_count.store(put_bytes, Ordering::Relaxed);
metric
.deletion_count
.store((rows - live).max(0), Ordering::Relaxed);
.store((puts - live).max(0), Ordering::Relaxed);
metric
.deletion_byte_count
.store(valid_bytes.saturating_sub(live_bytes), Ordering::Relaxed);
.store(put_bytes.saturating_sub(live_bytes), Ordering::Relaxed);
Ok(metric)
}
@@ -617,11 +612,11 @@ mod tests {
);
let m = SortedFileNeedleMap::open(&base, version()).unwrap();
// Every row counts, matching Go's FileCounter.
assert_eq!(m.file_count(), 5);
// Every put row counts, matching Go's FileCounter; the tombstone does not.
assert_eq!(m.file_count(), 4);
assert_eq!(m.content_size(), 100 + 200 + 150 + 300);
// Superseded row for key 1, plus both rows for the deleted key 2.
assert_eq!(m.deleted_count(), 3);
// Superseded row for key 1, plus the put of the deleted key 2.
assert_eq!(m.deleted_count(), 2);
assert_eq!(m.deleted_size(), 100 + 200);
assert_eq!(m.max_file_key(), NeedleId(3));
}
@@ -709,8 +704,8 @@ mod tests {
// already agree with it or the heartbeat jumps at every restart.
drop(m);
let reloaded = SortedFileNeedleMap::open(&base, version()).unwrap();
assert_eq!(reloaded.file_count(), 4);
assert_eq!(reloaded.deleted_count(), 2);
assert_eq!(reloaded.file_count(), 3);
assert_eq!(reloaded.deleted_count(), 1);
assert_eq!(reloaded.deleted_size(), 200);
assert_eq!(reloaded.content_size(), 600);
@@ -718,7 +713,7 @@ mod tests {
reloaded
.delete(NeedleId(2), Offset::from_actual_offset(16))
.unwrap();
assert_eq!((reloaded.file_count(), reloaded.deleted_count()), (4, 2));
assert_eq!((reloaded.file_count(), reloaded.deleted_count()), (3, 1));
}
#[test]
@@ -960,7 +955,7 @@ mod tests {
assert_eq!(
m.deleted_count(),
deleted_before + 2,
deleted_before + 1,
"a durable tombstone must move the deletion count"
);
assert_eq!(
@@ -979,7 +974,7 @@ mod tests {
.unwrap(),
None
);
assert_eq!(m.deleted_count(), deleted_before + 2);
assert_eq!(m.deleted_count(), deleted_before + 1);
assert_eq!(m.deleted_size(), bytes_before + 100);
}
+257 -24
View File
@@ -948,14 +948,39 @@ impl Store {
vol.read_needle_opt(n, read_deleted)
}
/// Resolve a needle read under this guard, to run after it is released.
/// Resolve a needle read with the store guard held only for the lookup:
/// the plan can wait on a vacuum commit's data-file lease, and waiting
/// under the guard would park store writers, and every reader queued
/// behind them, on that commit. A handle closed while the plan waited is
/// looked up again, a bounded number of times.
pub(crate) fn needle_read_plan(
&self,
store: &std::sync::RwLock<Store>,
vid: VolumeId,
id: NeedleId,
read_deleted: bool,
) -> Result<crate::storage::volume::NeedleReadPlan, VolumeError> {
let (_, vol) = self.find_volume(vid).ok_or(VolumeError::NotFound)?;
let find = || {
let store = store.read().unwrap();
store
.find_volume(vid)
.map(|(_, vol)| vol)
.ok_or(VolumeError::NotFound)
};
Self::plan_on_open_handle(find, id, read_deleted)
}
fn plan_on_open_handle(
mut find: impl FnMut() -> Result<Arc<Volume>, VolumeError>,
id: NeedleId,
read_deleted: bool,
) -> Result<crate::storage::volume::NeedleReadPlan, VolumeError> {
let mut vol = find()?;
for _ in 0..3 {
match vol.needle_read_plan(id, read_deleted) {
Err(_) if vol.is_closed() => vol = find()?,
plan => return plan,
}
}
vol.needle_read_plan(id, read_deleted)
}
@@ -1157,13 +1182,14 @@ impl Store {
/// Matches Go's Store.MountEcShards which mounts one shard at a time.
/// `source_disk_type` is the source volume's disk type from the
/// `VolumeEcShardsMount` RPC (#9423); pass `""` for non-RPC paths.
/// Returns the index of the disk the shard mounted on.
pub fn mount_ec_shard(
&mut self,
vid: VolumeId,
collection: &str,
shard_id: ShardId,
source_disk_type: &str,
) -> Result<(), VolumeError> {
) -> Result<usize, VolumeError> {
self.stamp_allocated_vid(vid);
// The .ecx may live on a different disk than the shard being mounted
// (ec.balance / ec.rebuild can spread shards across sibling disks), so
@@ -1176,7 +1202,7 @@ impl Store {
// next. Like Go, a NotFound just means "not this disk"; anything else
// is collected so an all-disks-fail error names every disk tried.
let mut failures: Vec<(String, VolumeError)> = Vec::new();
for loc in &mut self.locations {
for (disk, loc) in self.locations.iter_mut().enumerate() {
// Check if the shard file exists on this location
let shard = EcVolumeShard::new(&loc.directory, collection, vid, shard_id);
if !std::path::Path::new(&shard.file_name()).exists() {
@@ -1201,7 +1227,7 @@ impl Store {
&idx_dir,
source_disk_type,
) {
Ok(()) => return Ok(()),
Ok(()) => return Ok(disk),
Err(VolumeError::Io(e)) if e.kind() == io::ErrorKind::NotFound => {}
Err(e) => failures.push((loc.directory.clone(), e)),
}
@@ -1466,22 +1492,27 @@ impl Store {
Vec<master_pb::VolumeEcShardInformationMessage>,
Vec<master_pb::VolumeEcShardInformationMessage>,
) {
let (mut ec_shards, expired) = self.find_expired_ec_volumes();
let (deleted, still_held) = self.remove_expired_ec_volumes(expired);
let (mut ec_shards, expired, quarantined) = self.find_expired_ec_volumes();
let (mut deleted, still_held) = self.remove_expired_ec_volumes(expired);
ec_shards.extend(still_held);
deleted.extend(quarantined);
(ec_shards, deleted)
}
/// The read half of `delete_expired_ec_volumes`: the shards to report, and
/// the expired EC volumes, by disk index, for `remove_expired_ec_volumes`.
/// The read half of `delete_expired_ec_volumes`: the shards to report, the
/// expired EC volumes, by disk index, for `remove_expired_ec_volumes`, and
/// the shards of volumes this pass quarantined, which go out as deleted
/// once so the master drops them before the next full EC heartbeat.
pub fn find_expired_ec_volumes(
&self,
) -> (
Vec<master_pb::VolumeEcShardInformationMessage>,
Vec<(usize, VolumeId)>,
Vec<master_pb::VolumeEcShardInformationMessage>,
) {
let mut ec_shards = Vec::new();
let mut expired = Vec::new();
let mut quarantined_now = Vec::new();
for (disk_id, loc) in self.locations.iter().enumerate() {
for (vid, ec_vol) in loc.ec_volumes() {
@@ -1496,6 +1527,8 @@ impl Store {
io_count,
"ec volume quarantined after consecutive IO errors"
);
quarantined_now
.extend(ec_vol.to_volume_ec_shard_information_messages(disk_id as u32));
}
} else {
ec_shards
@@ -1504,7 +1537,7 @@ impl Store {
}
}
(ec_shards, expired)
(ec_shards, expired, quarantined_now)
}
/// The write half of `delete_expired_ec_volumes`: destroys each volume that
@@ -1774,7 +1807,7 @@ impl Store {
{
match self.begin_compact_volume(vid, preallocate)? {
Some(job) => job.run(progress_fn),
None => Ok(()),
None => Err(crate::storage::volume::compacting_error(vid)),
}
}
@@ -3039,6 +3072,7 @@ mod tests {
#[test]
fn test_maybe_adjust_volume_max_reserves_plain_volumes_while_preallocating() {
let _g = FAKE_DISK_LOCK.lock().unwrap();
let tmp = TempDir::new().unwrap();
let dir = tmp.path().to_str().unwrap();
@@ -3047,7 +3081,7 @@ mod tests {
.add_location(
dir,
dir,
2,
5,
DiskType::HardDrive,
MinFreeSpace::Percent(1.0),
Vec::new(),
@@ -3064,15 +3098,13 @@ mod tests {
)
.unwrap();
// Room for 2.5 volumes: the plain volume claims one, leaving one more slot.
let (_, free) = crate::storage::disk_location::get_disk_stats(dir);
let volume_size_limit = (free as f64 / 2.5) as u64;
if volume_size_limit == 0 {
return; // cannot determine free disk space
}
store
.volume_size_limit
.store(volume_size_limit, Ordering::Relaxed);
// Room for 2.5 volumes above the 1% floor: the plain volume claims one,
// leaving one more slot.
let limit = 1024 * MIB;
FAKE_ALL.store(100 * limit, Ordering::Relaxed);
FAKE_FREE.store(limit + limit * 5 / 2, Ordering::Relaxed);
store.disk_stats_of = fake_disk_stats;
store.volume_size_limit.store(limit, Ordering::Relaxed);
store.locations[0].original_max_volume_count = 0;
store.set_preallocate(true);
@@ -3385,6 +3417,207 @@ mod tests {
assert_eq!(v.dat_file_size().unwrap(), volume_size);
}
/// A volume `vid` holding needle 1, compacted and ready to commit, behind
/// the store lock the server's handlers use.
fn store_ready_to_commit(
dir: &str,
vid: VolumeId,
kind: NeedleMapKind,
) -> Arc<std::sync::RwLock<Store>> {
let mut store = make_test_store(&[dir]);
store.needle_map_kind = kind;
store
.add_volume(vid, DiskType::HardDrive, &VolumeSpec::default())
.unwrap();
write_test_needle(&mut store, vid, 1, b"live needle");
store.compact_volume(vid, 0, 0, |_| true).unwrap();
Arc::new(std::sync::RwLock::new(store))
}
/// Commits as the vacuum-commit RPC does: the volume is looked up under
/// the store guard, which is dropped before the commit takes its locks.
/// The result comes back on a channel, so a deadlock fails a test by
/// timing out instead of hanging it.
fn commit_like_the_rpc(
store: Arc<std::sync::RwLock<Store>>,
vid: VolumeId,
) -> std::sync::mpsc::Receiver<Result<(), VolumeError>> {
let (committed, result) = std::sync::mpsc::channel();
std::thread::spawn(move || {
let (_, vol) = store.read().unwrap().find_volume(vid).unwrap();
let _sync_guard = vol.data_file_sync_lock().lock().unwrap();
let _lease = vol.instance().write_lock();
let _ = committed.send(vol.commit_compact_held());
});
result
}
/// Holds the next commit of `vid` inside its swap window, where the needle
/// map is closed, until the returned sender is used.
fn hold_commit_in_swap_window(
vid: VolumeId,
) -> (
crate::storage::volume::commit_swap_hook::Registration,
std::sync::mpsc::Receiver<()>,
std::sync::mpsc::Sender<()>,
) {
let (entered, in_window) = std::sync::mpsc::channel();
let (release, released) = std::sync::mpsc::channel::<()>();
let released = std::sync::Mutex::new(released);
let entered = std::sync::Mutex::new(entered);
let hook = crate::storage::volume::commit_swap_hook::register(vid, move || {
entered.lock().unwrap().send(()).unwrap();
released.lock().unwrap().recv().unwrap();
});
(hook, in_window, release)
}
/// Plans and reads needle 1 of `vid` on another thread, and returns once
/// the plan holds the volume handle, so a timed check cannot pass on a
/// late start.
fn plan_in_background(
store: &Arc<std::sync::RwLock<Store>>,
vid: VolumeId,
) -> std::sync::mpsc::Receiver<Result<Vec<u8>, VolumeError>> {
let (_, vol) = store.read().unwrap().find_volume(vid).unwrap();
let handles = Arc::strong_count(&vol);
let (planned, plan) = std::sync::mpsc::channel();
let reader = store.clone();
std::thread::spawn(move || {
let mut n = Needle {
id: NeedleId(1),
cookie: Cookie(1),
..Needle::default()
};
let read = Store::needle_read_plan(&reader, vid, NeedleId(1), false)
.and_then(|mut plan| plan.read_full(&mut n))
.map(|()| n.data);
let _ = planned.send(read);
});
let deadline = std::time::Instant::now() + std::time::Duration::from_secs(10);
while Arc::strong_count(&vol) == handles {
assert!(
std::time::Instant::now() < deadline,
"the plan never looked up the volume"
);
std::thread::sleep(std::time::Duration::from_millis(1));
}
plan
}
/// A GET planned while a vacuum commit has the needle map closed waits for
/// the commit, as Go's readNeedle does, instead of answering NotFound for a
/// live needle.
fn check_read_plan_waits_out_a_commit_swap_window(kind: NeedleMapKind, vid: VolumeId) {
let tmp = TempDir::new().unwrap();
let store = store_ready_to_commit(tmp.path().to_str().unwrap(), vid, kind);
let (_hook, in_window, release) = hold_commit_in_swap_window(vid);
let commit = commit_like_the_rpc(store.clone(), vid);
in_window
.recv_timeout(std::time::Duration::from_secs(10))
.expect("the commit never reached its swap window");
let plan = plan_in_background(&store, vid);
if let Ok(early) = plan.recv_timeout(std::time::Duration::from_millis(300)) {
release.send(()).unwrap();
panic!("planned during the swap window instead of waiting: {early:?}");
}
release.send(()).unwrap();
commit
.recv_timeout(std::time::Duration::from_secs(10))
.expect("the commit never finished")
.unwrap();
let after = plan
.recv_timeout(std::time::Duration::from_secs(10))
.expect("the plan still waits after the commit finished");
assert_eq!(after.unwrap(), b"live needle");
}
#[test]
fn test_read_plan_waits_out_a_commit_swap_window_in_memory() {
check_read_plan_waits_out_a_commit_swap_window(NeedleMapKind::InMemory, VolumeId(81));
}
#[test]
fn test_read_plan_waits_out_a_commit_swap_window_redb() {
check_read_plan_waits_out_a_commit_swap_window(NeedleMapKind::Redb, VolumeId(83));
}
/// A plan on a handle an unmount closed looks the volume up again and
/// reads the remounted volume.
#[test]
fn test_read_plan_retries_a_handle_closed_by_a_remount() {
let tmp = TempDir::new().unwrap();
let vid = VolumeId(1);
let mut store = make_test_store(&[tmp.path().to_str().unwrap()]);
store
.add_volume(vid, DiskType::HardDrive, &VolumeSpec::default())
.unwrap();
write_test_needle(&mut store, vid, 1, b"live needle");
let (_, stale) = store.find_volume(vid).unwrap();
assert!(store.unmount_volume(vid).unwrap());
store.mount_volume(vid, "", DiskType::HardDrive).unwrap();
let mut stale = Some(stale);
let mut plan = Store::plan_on_open_handle(
|| match stale.take() {
Some(vol) => Ok(vol),
None => store
.find_volume(vid)
.map(|(_, vol)| vol)
.ok_or(VolumeError::NotFound),
},
NeedleId(1),
false,
)
.unwrap();
let mut n = Needle {
id: NeedleId(1),
cookie: Cookie(1),
..Needle::default()
};
plan.read_full(&mut n).unwrap();
assert_eq!(n.data, b"live needle");
}
/// A plan waiting for a commit holds no store guard, so a store writer
/// finishes while the commit is still in its swap window.
#[test]
fn test_read_plan_waiting_on_a_commit_does_not_block_a_store_writer() {
let tmp = TempDir::new().unwrap();
let vid = VolumeId(82);
let store =
store_ready_to_commit(tmp.path().to_str().unwrap(), vid, NeedleMapKind::InMemory);
let (_hook, in_window, release) = hold_commit_in_swap_window(vid);
let commit = commit_like_the_rpc(store.clone(), vid);
in_window
.recv_timeout(std::time::Duration::from_secs(10))
.expect("the commit never reached its swap window");
let plan = plan_in_background(&store, vid);
let (wrote, written) = std::sync::mpsc::channel();
let writer = store.clone();
std::thread::spawn(move || {
let _guard = writer.write().unwrap();
let _ = wrote.send(());
});
let writer_done = written.recv_timeout(std::time::Duration::from_secs(10));
let planned_early = plan.try_recv().is_ok();
release.send(()).unwrap();
writer_done.expect("a store writer waited on the commit");
assert!(!planned_early, "planned during the swap window");
let committed = commit
.recv_timeout(std::time::Duration::from_secs(10))
.expect("the commit never finished");
assert!(committed.is_ok(), "{committed:?}");
let after = plan
.recv_timeout(std::time::Duration::from_secs(10))
.expect("the plan still waits after the commit finished");
assert_eq!(after.unwrap(), b"live needle");
}
fn write_test_needle(store: &mut Store, vid: VolumeId, id: u64, data: &[u8]) {
let mut n = Needle {
id: NeedleId(id),
@@ -3438,8 +3671,8 @@ mod tests {
assert!(store.cleanup_compact_volume(vid).is_err());
let (_, v) = store.find_volume(vid).unwrap();
assert!(v.relocate_index_to(other.path().to_str().unwrap()).is_err());
// Go parity: a commit that finds the volume compacting is a no-op.
store.commit_compact_volume(vid).unwrap();
assert!(store.commit_compact_volume(vid).is_err());
assert!(store.compact_volume(vid, 0, 0, |_| true).is_err());
let (_, v) = store.find_volume(vid).expect("still mounted");
assert!(v.is_compacting());
File diff suppressed because it is too large. Load diff
+779 -9
View File
File diff suppressed because it is too large. Load diff
+1 -1
View File
@@ -6,7 +6,7 @@
# sorting jobs share, so that two of them cannot drift on what an order means.
[workspace]
resolver = "2"
members = ["crates/core", "crates/lance", "crates/sort"]
members = ["crates/core", "crates/lance", "crates/sort", "crates/sw-client"]
[workspace.package]
version = "0.1.0"
+22 -13
View File
@@ -35,7 +35,7 @@ binary for the runner's platform, resolved from the version in `Cargo.lock`.
## Running
cargo run -p weed-lance-worker -- --admin 127.0.0.1:12535
cargo run -p weed-lance-worker -- --admin 127.0.0.1:12535 --masters 127.0.0.1:9333
The admin's *HTTP* address is what an operator has; the gRPC port is derived
from it the way the Go side does. Dialling the HTTP port fails as "frame with
@@ -49,7 +49,7 @@ Released builds do not need a toolchain. The worker ships inside the SeaweedFS
image, beside the Rust volume server, under the verb that mirrors
`volume-rust`:
docker run chrislusf/seaweedfs worker-rust --admin admin:12535
docker run chrislusf/seaweedfs worker-rust --admin admin:12535 --masters master:9333
and as `weed-worker_linux_{amd64,arm64}.tar.gz` on each GitHub release. Both are
linux amd64/arm64 only — lance, arrow and datafusion make every extra target an
@@ -59,7 +59,7 @@ entrypoint says so rather than failing as "not found".
## Metrics
cargo run -p weed-lance-worker -- --admin 127.0.0.1:12535 --metrics-port 9328
cargo run -p weed-lance-worker -- --admin 127.0.0.1:12535 --masters 127.0.0.1:9333 --metrics-port 9328
Serves `/health`, `/ready` and `/metrics` on that port, the same three the Go
worker serves under `weed worker -metricsPort`, so one scrape config covers
@@ -89,16 +89,22 @@ running but is not going to do anything.
## Credentials
The worker holds none. It asks the namespace to describe a table with
`vend_credentials` and hands the `storage_options` that come back to lance. A
gateway without STS configured vends no credentials at all, so `--access-key`
and `--secret-key` supply a fallback; anything the namespace does vend wins over
them.
The worker holds no S3 credentials. The namespace tells it where a table is;
its files are read and written on the cluster's filers and volume servers,
which `--masters` names the masters of (`--filer-group` picks the filers). What
that takes comes from the same security.toml the weed processes use:
`grpc.client` for the filers, `https.client` for volume servers under HTTPS,
`jwt.signing.read.key` where volume reads are signed, and `s3.sse.kek` or
`s3.sse.key` where a bucket encrypts with SSE-S3. Chunks are encrypted for the
volume servers when the filer is (`weed filer -encryptVolumeData`), as the S3
gateway's are.
In a vector bucket's `vector` namespace the gateway vends write credentials only
to a caller that proves it holds `jwt.filer_signing.key`. Pass that key with
`--filer-signing-key` so compaction, cleanup and index builds can rewrite
vector indexes.
The namespace authenticates the worker by the cluster token it derives from
`jwt.filer_signing.key`; the same key signs the admin stream when admin requires
it. The worker reads it from where `weed worker` does: `WEED_JWT_FILER_SIGNING_KEY`,
or `[jwt.filer_signing]` in the security.toml found in the working directory,
`~/.seaweedfs`, `/usr/local/etc/seaweedfs` or `/etc/seaweedfs`.
`--filer-signing-key` overrides both.
## State
@@ -110,7 +116,10 @@ All three jobs are implemented and tested end to end against a live gateway:
`cargo test -p weed-lance-worker` runs them when `WEED_LANCE_NAMESPACE` names a
live namespace and skips otherwise, the way the Go integration tests skip
without Docker. Each test seeds the table it needs, including building a vector
without Docker. A live namespace needs `WEED_MASTERS` naming its cluster's
masters, and `WEED_JWT_FILER_SIGNING_KEY` (or security.toml) where the
namespace authenticates cluster processes; the tests fail without the first
and are refused without the second. Each test seeds the table it needs, including building a vector
index and then appending rows outside it, so a run does not depend on what the
previous one left behind — the first version of these did, and quietly stopped
testing anything once it had done its job.
+5
View File
@@ -16,6 +16,8 @@ seaweed-common = { path = "../../../seaweed-common" }
# admin token can sign with the same HMAC the Go worker uses.
aws-lc-rs = "1"
base64 = "0.22"
# The same requirement seaweed-volume states for its security.toml reader.
toml = "0.8"
anyhow.workspace = true
async-trait.workspace = true
prost.workspace = true
@@ -28,6 +30,9 @@ axum.workspace = true
prometheus.workspace = true
tracing.workspace = true
[dev-dependencies]
tempfile = "3"
[build-dependencies]
tonic-prost-build.workspace = true
# Ships protoc with the build so neither CI nor a developer needs a system
+1
View File
@@ -11,6 +11,7 @@ pub mod config;
pub mod config_form;
pub mod metrics;
pub mod registry;
pub mod security_config;
pub mod senders;
pub mod stream;
pub mod token;
@@ -0,0 +1,249 @@
//! The admin-stream signing key, read from where the Go worker reads it.
//!
//! `weed worker` takes `jwt.filer_signing` from security.toml, with the
//! `WEED_JWT_FILER_SIGNING_*` environment variables overriding the file the
//! way viper does. A Rust worker started beside it, from the same image with
//! the same mounts, has to find the key in the same places or it connects
//! with no token and admin turns it away.
use std::ffi::OsString;
use std::path::{Path, PathBuf};
use seaweed_common::security_config as location;
/// `jwt.filer_signing` as the Go worker resolves it.
#[derive(Clone, Debug, PartialEq, Eq)]
pub struct FilerSigning {
pub key: String,
/// Zero mints a token without an expiry, the Go default.
pub expires_after_seconds: u64,
/// Where the key came from, for the startup log.
pub source: String,
}
/// The key from the process environment and the discovered security.toml, or
/// None when neither configures one.
pub fn filer_signing() -> Option<FilerSigning> {
let env = |key: &str| std::env::var_os(key);
filer_signing_in(&env, std::env::current_dir().ok().as_deref())
}
fn filer_signing_in(
env: &dyn Fn(&str) -> Option<OsString>,
current_dir: Option<&Path>,
) -> Option<FilerSigning> {
let file = location::find(current_dir, location::home_dir(env).as_deref());
filer_signing_from(env, file.as_deref())
}
fn filer_signing_from(
env: &dyn Fn(&str) -> Option<OsString>,
file: Option<&Path>,
) -> Option<FilerSigning> {
// Viper treats an empty environment value as unset.
let env_string = |key: &str| {
env(key)
.and_then(|v| v.into_string().ok())
.filter(|v| !v.is_empty())
};
let (file_key, file_expires) = file.map(read_file).unwrap_or_default();
let (key, source) = match env_string("WEED_JWT_FILER_SIGNING_KEY") {
Some(key) => (key, "WEED_JWT_FILER_SIGNING_KEY".to_string()),
None => (file_key?, file.map(PathBuf::from)?.display().to_string()),
};
if key.is_empty() {
return None;
}
let expires_after_seconds = env_string("WEED_JWT_FILER_SIGNING_EXPIRES_AFTER_SECONDS")
.and_then(|v| v.parse().ok())
.or(file_expires)
.unwrap_or(0);
Some(FilerSigning {
key,
expires_after_seconds,
source,
})
}
/// `[jwt.filer_signing]` key and expires_after_seconds from one file. A file
/// that cannot be read or parsed configures nothing, as it does for viper,
/// which also logs a warning.
fn read_file(path: &Path) -> (Option<String>, Option<u64>) {
// The message alone: the full parse error quotes the offending line,
// which can hold a key.
let parsed = std::fs::read_to_string(path)
.map_err(|e| e.to_string())
.and_then(|content| {
content
.parse::<toml::Table>()
.map_err(|e| e.message().to_string())
});
let table = match parsed {
Ok(table) => table,
Err(error) => {
tracing::warn!(path = %path.display(), %error, "skipping unreadable security.toml");
return (None, None);
}
};
let section = table
.get("jwt")
.and_then(|jwt| jwt.get("filer_signing"))
.and_then(|section| section.as_table());
let Some(section) = section else {
return (None, None);
};
let key = section
.get("key")
.and_then(|v| v.as_str())
.map(str::to_string);
// Viper's GetInt also reads a quoted number and truncates a float.
let expires = section.get("expires_after_seconds").and_then(|v| match v {
toml::Value::Integer(n) => u64::try_from(*n).ok(),
toml::Value::Float(f) if f.is_finite() && *f >= 0.0 => Some(*f as u64),
toml::Value::String(s) => s.trim().parse().ok(),
_ => None,
});
(key, expires)
}
#[cfg(test)]
mod tests {
use super::*;
use std::collections::HashMap;
fn env_of(pairs: &[(&str, &str)]) -> impl Fn(&str) -> Option<OsString> {
let map: HashMap<String, OsString> = pairs
.iter()
.map(|(k, v)| (k.to_string(), OsString::from(v)))
.collect();
move |key: &str| map.get(key).cloned()
}
fn write(dir: &tempfile::TempDir, content: &str) -> PathBuf {
let path = dir.path().join(location::SECURITY_CONFIG_FILE_NAME);
std::fs::write(&path, content).unwrap();
path
}
#[test]
fn reads_the_section_the_go_worker_reads() {
let dir = tempfile::TempDir::new().unwrap();
let path = write(
&dir,
r#"
[jwt.signing]
key = "volume-key"
[jwt.filer_signing]
key = "cluster-key"
expires_after_seconds = 300
[jwt.filer_signing.read]
key = "read-key"
"#,
);
let got = filer_signing_from(&env_of(&[]), Some(&path)).unwrap();
assert_eq!(got.key, "cluster-key");
assert_eq!(got.expires_after_seconds, 300);
assert_eq!(got.source, path.display().to_string());
}
#[test]
fn quoted_expiry_reads_as_a_number() {
let dir = tempfile::TempDir::new().unwrap();
let path = write(
&dir,
"[jwt.filer_signing]\nkey = \"k\"\nexpires_after_seconds = \"300\"\n",
);
let got = filer_signing_from(&env_of(&[]), Some(&path)).unwrap();
assert_eq!(got.expires_after_seconds, 300);
}
#[test]
fn float_expiry_truncates_to_seconds() {
let dir = tempfile::TempDir::new().unwrap();
let path = write(
&dir,
"[jwt.filer_signing]\nkey = \"k\"\nexpires_after_seconds = 300.9\n",
);
let got = filer_signing_from(&env_of(&[]), Some(&path)).unwrap();
assert_eq!(got.expires_after_seconds, 300);
}
#[test]
fn environment_overrides_the_file() {
let dir = tempfile::TempDir::new().unwrap();
let path = write(
&dir,
"[jwt.filer_signing]\nkey = \"file-key\"\nexpires_after_seconds = 5\n",
);
let env = env_of(&[
("WEED_JWT_FILER_SIGNING_KEY", "env-key"),
("WEED_JWT_FILER_SIGNING_EXPIRES_AFTER_SECONDS", "60"),
]);
let got = filer_signing_from(&env, Some(&path)).unwrap();
assert_eq!(got.key, "env-key");
assert_eq!(got.expires_after_seconds, 60);
assert_eq!(got.source, "WEED_JWT_FILER_SIGNING_KEY");
}
#[test]
fn environment_key_alone_is_enough() {
let got =
filer_signing_from(&env_of(&[("WEED_JWT_FILER_SIGNING_KEY", "k")]), None).unwrap();
assert_eq!(got.key, "k");
assert_eq!(got.expires_after_seconds, 0);
}
#[test]
fn empty_environment_values_fall_back_to_the_file() {
let dir = tempfile::TempDir::new().unwrap();
let path = write(
&dir,
"[jwt.filer_signing]\nkey = \"file-key\"\nexpires_after_seconds = 5\n",
);
let env = env_of(&[
("WEED_JWT_FILER_SIGNING_KEY", ""),
("WEED_JWT_FILER_SIGNING_EXPIRES_AFTER_SECONDS", ""),
]);
let got = filer_signing_from(&env, Some(&path)).unwrap();
assert_eq!(got.key, "file-key");
assert_eq!(got.expires_after_seconds, 5);
assert_eq!(got.source, path.display().to_string());
}
#[test]
fn discovers_the_file_in_the_working_directory_then_home() {
let work = tempfile::TempDir::new().unwrap();
let home = tempfile::TempDir::new().unwrap();
std::fs::create_dir(home.path().join(".seaweedfs")).unwrap();
let in_home = home
.path()
.join(".seaweedfs")
.join(location::SECURITY_CONFIG_FILE_NAME);
std::fs::write(&in_home, "[jwt.filer_signing]\nkey = \"home-key\"\n").unwrap();
let vars = [("HOME", home.path().to_str().unwrap())];
let env = env_of(&vars);
let got = filer_signing_in(&env, Some(work.path())).unwrap();
assert_eq!(got.key, "home-key");
assert_eq!(got.source, in_home.display().to_string());
write(&work, "[jwt.filer_signing]\nkey = \"work-key\"\n");
assert_eq!(
filer_signing_in(&env, Some(work.path())).unwrap().key,
"work-key"
);
}
#[test]
fn nothing_configured_is_none() {
let dir = tempfile::TempDir::new().unwrap();
let other_section = write(&dir, "[jwt.signing]\nkey = \"volume-key\"\n");
assert_eq!(filer_signing_from(&env_of(&[]), Some(&other_section)), None);
assert_eq!(filer_signing_from(&env_of(&[]), None), None);
let empty = env_of(&[("WEED_JWT_FILER_SIGNING_KEY", "")]);
assert_eq!(filer_signing_from(&empty, None), None);
}
}
+15 -2
View File
@@ -3,6 +3,9 @@ use std::time::{SystemTime, UNIX_EPOCH};
use aws_lc_rs::hmac;
use base64::{engine::general_purpose::URL_SAFE_NO_PAD, Engine};
/// 9999-12-31T23:59:59Z; golang-jwt reads `exp` through float64 and time.Unix.
const MAX_EXP_SECS: u64 = 253_402_300_799;
/// Mints the HS256 Bearer token the admin control stream requires when
/// jwt.filer_signing.key is configured, matching the SeaweedFilerAdminClaims
/// the Go worker sends: registered claims only, no session id.
@@ -15,7 +18,8 @@ pub fn admin_token(signing_key: &[u8], expires_after_secs: u64) -> Option<String
.duration_since(UNIX_EPOCH)
.map(|d| d.as_secs())
.unwrap_or_default()
+ expires_after_secs;
.saturating_add(expires_after_secs)
.min(MAX_EXP_SECS);
format!("{{\"exp\":{exp}}}")
} else {
"{}".to_string()
@@ -42,7 +46,7 @@ pub fn derive_key(signing_key: &[u8], purpose: &str) -> Vec<u8> {
#[cfg(test)]
mod tests {
use super::{admin_token, derive_key};
use super::{admin_token, derive_key, MAX_EXP_SECS};
#[test]
fn derive_key_matches_the_go_derivation() {
@@ -75,4 +79,13 @@ mod tests {
let claims = URL_SAFE_NO_PAD.decode(payload).unwrap();
assert_eq!(claims, b"{}");
}
#[test]
fn huge_expiry_is_capped() {
let token = admin_token(b"secret", u64::MAX).unwrap();
let payload = token.split('.').nth(1).unwrap();
use base64::{engine::general_purpose::URL_SAFE_NO_PAD, Engine};
let claims = URL_SAFE_NO_PAD.decode(payload).unwrap();
assert_eq!(claims, format!("{{\"exp\":{MAX_EXP_SECS}}}").as_bytes());
}
}
+41 -1
View File
@@ -22,11 +22,14 @@ path = "src/main.rs"
seaweed-common = { path = "../../../seaweed-common" }
seaweed-worker-core = { path = "../core" }
seaweed-worker-sort = { path = "../sort" }
sw-client = { path = "../sw-client" }
prometheus.workspace = true
# Only the S3 backend: the other object stores lance enables by default are
# build time this worker never spends.
lance = { version = "10", default-features = false, features = ["aws"] }
lance-index = "10"
lance-linalg = "10"
roaring = "0.11"
# For the manifest's own fragment type, which the sort marker summarises.
lance-table = "10"
# For the execution options the sort needs; the scanner's own stream helper
@@ -45,14 +48,51 @@ serde_json = "1"
tokio.workspace = true
tracing.workspace = true
tracing-subscriber.workspace = true
# Reads straight from the filer and volume servers: object_store's trait,
# the gateway's object attributes and ETags, and a read JWT per volume read.
object_store = "0.13"
bytes = "1"
moka = { version = "0.12", features = ["sync"] }
jsonwebtoken = { version = "10", features = ["rust_crypto"] }
md-5 = "0.10"
hex = "0.4"
base64 = "0.22"
# The S3 gateway's SSE-S3 envelope: AES-256-GCM wraps keys, AES-256-CTR wraps
# bytes, HMAC-SHA256 commits them; the filer's volume encryption is AES-256-GCM.
aes = "0.8"
ctr = "0.9"
aes-gcm = "0.10"
hmac = "0.12"
sha2 = "0.10"
hkdf = "0.12"
flate2 = "1"
zstd = "0.13"
# security.toml and the gRPC TLS weed processes use with it: the client
# connector with the server allowlist, and the acceptor the engine serves with.
toml = "0.8"
rustls = "0.23"
tokio-rustls = "0.26"
x509-parser = "0.16"
tower = "0.5"
hyper-util = { version = "0.1", features = ["tokio"] }
http = "1"
tonic.workspace = true
tonic-prost.workspace = true
prost.workspace = true
tokio-stream.workspace = true
[build-dependencies]
tonic-prost-build.workspace = true
protoc-bin-vendored = "3"
[dev-dependencies]
tokio = { workspace = true }
tokio = { workspace = true, features = ["test-util"] }
tonic.workspace = true
arrow-array = "58"
arrow-schema = "58"
arrow-cast = "58"
lance-linalg = "10"
tempfile = "3"
[lints]
workspace = true
+29
View File
@@ -0,0 +1,29 @@
fn main() -> Result<(), Box<dyn std::error::Error>> {
// Use the protoc that ships with protoc-bin-vendored rather than a system
// one, so the build needs no package manager and always sees the same
// version. An explicit PROTOC still wins, for packagers supplying their own
// and for the lance crates, whose own build scripts read the same variable.
if std::env::var_os("PROTOC").is_none() {
// SAFETY: a build script's main runs single-threaded before anything
// else in this process, so no other thread can be reading the
// environment concurrently.
unsafe {
std::env::set_var("PROTOC", protoc_bin_vendored::protoc_bin_path()?);
}
}
// The master's ListClusterNodes, for the filers; master.proto brings the
// two it imports.
tonic_prost_build::configure()
.build_server(false)
.build_client(true)
.compile_protos(
&[
"../../../weed/pb/master.proto",
"../../../weed/pb/volume_server.proto",
"../../../weed/pb/remote.proto",
],
&["../../../weed/pb/"],
)?;
println!("cargo:rerun-if-changed=../../../weed/pb/master.proto");
Ok(())
}
@@ -67,7 +67,6 @@ struct ListTablesResponse {
pub struct NamespaceClient {
base_url: String,
http: reqwest::Client,
api_key: Option<String>,
signing_key: Option<SigningKey>,
}
@@ -104,29 +103,10 @@ impl NamespaceClient {
Self {
base_url: base_url.into().trim_end_matches('/').to_string(),
http,
api_key: None,
signing_key: None,
}
}
/// The catalog accepts an S3 credential as `x-api-key`; where the worker
/// already holds one for storage it authenticates the namespace calls too.
pub fn with_fallback(mut self, fallback: &crate::dataset::FallbackOptions) -> Self {
if let (Some(access), Some(secret)) = (
fallback.get("aws_access_key_id"),
fallback.get("aws_secret_access_key"),
) {
if !protected_transport(&self.base_url) {
tracing::warn!(
"catalog url {} exposes the x-api-key credential in cleartext",
self.base_url
);
}
self.api_key = Some(format!("{access}:{secret}"));
}
self
}
pub fn with_signing_key(mut self, signing_key: Option<&SigningKey>) -> Self {
if signing_key.is_some() && !protected_transport(&self.base_url) {
tracing::warn!(
@@ -139,10 +119,6 @@ impl NamespaceClient {
}
fn authed(&self, request: reqwest::RequestBuilder) -> reqwest::RequestBuilder {
let request = match &self.api_key {
Some(key) => request.header("x-api-key", key),
None => request,
};
match self.signing_key.as_ref().and_then(SigningKey::token) {
Some(token) => request.header(CLUSTER_TOKEN_HEADER, token),
None => request,
+293
View File
@@ -0,0 +1,293 @@
//! The cluster as a Lance process joins it: the masters dialed as weed
//! clients dial them, the filers they list, and the storage connection built
//! from them and security.toml that the engine, the worker and the CLI share.
use std::sync::Arc;
use std::time::Duration;
use anyhow::{Context, Result};
use tonic::transport::{Channel, Endpoint};
use crate::direct_read::{Connection, DirectReads, FilerDialer};
use crate::grpc_tls;
use crate::master_pb::{ListClusterNodesRequest, seaweed_client::SeaweedClient};
use crate::security::{GrpcTls, Security};
/// Masters are configured by their HTTP address; gRPC lives +10000 like the
/// weed conventions (server_address.ToGrpcAddress).
// A master address may carry an explicit gRPC port as `host:http.grpc` —
// use it directly; otherwise the gRPC port is http + 10000.
pub fn grpc_addr(master: &str) -> Result<String> {
let (scheme, rest) = master.split_once("://").unwrap_or(("http", master));
if let Some((host, port)) = rest.rsplit_once(':') {
if let Some((_, grpc)) = port.split_once('.') {
if let Ok(grpc) = grpc.parse::<u64>() {
let grpc = u16::try_from(grpc)
.with_context(|| format!("master {master}: gRPC port {grpc} is past 65535"))?;
return Ok(format!("{scheme}://{host}:{grpc}"));
}
}
if let Ok(port) = port.parse::<u64>() {
let grpc = port
.checked_add(10000)
.and_then(|grpc| u16::try_from(grpc).ok())
.with_context(|| {
format!(
"master {master}: gRPC port {port}+10000 is past 65535; \
give it explicitly as {host}:{port}.<grpc port>"
)
})?;
return Ok(format!("{scheme}://{host}:{grpc}"));
}
}
Ok(format!("{scheme}://{rest}"))
}
/// Bounds the credential reads and the connect, TLS handshake included, and
/// then the master's first answer, the 5s wdclient gives a master call; a
/// black-holed or silent master otherwise holds up the other masters.
pub const CONNECT_TIMEOUT: Duration = Duration::from_secs(5);
/// A channel to the master's gRPC, over TLS with `grpc.client`'s credentials
/// when the cluster has them.
pub async fn connect(master: &str, tls: Option<&GrpcTls>) -> Result<Channel> {
let addr = grpc_addr(master)?;
tokio::time::timeout(CONNECT_TIMEOUT, async {
Ok(match tls {
None => Endpoint::new(addr)?.connect().await?,
// The connector does the TLS; tonic adds its own only for an
// https URI, which it refuses without a tls_config.
Some(tls) => {
Endpoint::new(addr.replacen("https://", "http://", 1))?
.connect_with_connector(grpc_tls::connector(tls.clone()))
.await?
}
})
})
.await
.context("master did not complete the connection")?
}
/// The filers of `filer_group` registered with the masters, as gRPC
/// `host:port`s, in registration order; the masters are all asked, since a
/// follower may know fewer than the leader. An error means no master answered.
pub async fn list_filers(
masters: &[String],
filer_group: &str,
tls: Option<&GrpcTls>,
) -> Result<Vec<String>> {
let mut filers = Vec::new();
let mut answered = false;
let mut last = None;
for master in masters {
match filers_of(master, filer_group, tls).await {
Ok(found) => {
answered = true;
for filer in found {
if !filers.contains(&filer) {
filers.push(filer);
}
}
}
Err(err) => {
tracing::warn!("master {master}: listing filers: {err:#}");
last = Some(err);
}
}
}
match (answered, last) {
(false, Some(err)) => Err(err),
(false, None) => Err(anyhow::anyhow!("no master given")),
_ => Ok(filers),
}
}
async fn filers_of(master: &str, filer_group: &str, tls: Option<&GrpcTls>) -> Result<Vec<String>> {
let channel = connect(master, tls).await?;
let request = ListClusterNodesRequest {
client_type: "filer".into(),
filer_group: filer_group.into(),
..Default::default()
};
let nodes = tokio::time::timeout(
CONNECT_TIMEOUT,
SeaweedClient::new(channel).list_cluster_nodes(request),
)
.await
.context("master did not answer")??
.into_inner()
.cluster_nodes;
Ok(nodes
.iter()
.filter_map(|node| match filer_grpc_addr(&node.address) {
Ok(addr) => Some(addr),
Err(err) => {
tracing::warn!("master {master} lists filer {:?}: {err:#}", node.address);
None
}
})
.collect())
}
/// A filer registers as `host:port.grpc`, or `host:port` with its gRPC at
/// port + 10000.
pub fn filer_grpc_addr(address: &str) -> Result<String> {
let addr = grpc_addr(address)?;
Ok(addr
.split_once("://")
.map_or(addr.as_str(), |(_, rest)| rest)
.to_string())
}
/// How often the masters are asked for the filers.
const FILER_REFRESH: Duration = Duration::from_secs(30);
/// How long startup waits for a filer to be registered.
const FILER_WAIT: Duration = Duration::from_secs(120);
/// The filers the masters list, waiting for the first to register.
async fn discover_filers(
masters: &[String],
filer_group: &str,
tls: Option<&GrpcTls>,
) -> Result<Vec<String>> {
let deadline = tokio::time::Instant::now() + FILER_WAIT;
loop {
match list_filers(masters, filer_group, tls).await {
Ok(filers) if !filers.is_empty() => return Ok(filers),
Ok(_) => tracing::warn!("no filer is registered with the masters yet"),
Err(err) => tracing::warn!("listing filers: {err:#}"),
}
if tokio::time::Instant::now() >= deadline {
anyhow::bail!("no filer registered with the masters within {FILER_WAIT:?}");
}
tokio::time::sleep(Duration::from_secs(3)).await;
}
}
async fn follow_filers(
reads: DirectReads,
masters: Vec<String>,
filer_group: String,
tls: Option<GrpcTls>,
) {
loop {
tokio::time::sleep(FILER_REFRESH).await;
match list_filers(&masters, &filer_group, tls.as_ref()).await {
Ok(filers) if !filers.is_empty() => {
if let Err(err) = reads.set_filers(&filers) {
tracing::warn!("filers {}: {err:#}", filers.join(","));
}
}
// Keep the last known filers while the masters list none.
Ok(_) => tracing::warn!("the masters list no filer; keeping the known ones"),
Err(err) => tracing::warn!("listing filers: {err:#}"),
}
}
}
/// The comma-separated masters, each with a valid gRPC address.
pub fn master_addresses(masters: &str) -> Result<Vec<String>> {
let masters: Vec<String> = masters
.split(',')
.map(str::trim)
.filter(|s| !s.is_empty())
.map(str::to_string)
.collect();
if masters.is_empty() {
anyhow::bail!("the master list names no master address");
}
for master in &masters {
grpc_addr(master)?;
}
Ok(masters)
}
/// What the cluster's storage is reached with: the filers the masters list,
/// dialed as the masters are, and the keys security.toml holds.
pub struct Cluster {
pub masters: Vec<String>,
pub filer_group: String,
/// `grpc.client`'s credentials for the masters, when the cluster has them.
pub master_tls: Option<GrpcTls>,
}
impl Cluster {
pub fn new(masters: &str, filer_group: &str, security: &Security) -> Result<Self> {
Ok(Self {
masters: master_addresses(masters)?,
filer_group: filer_group.to_string(),
master_tls: security.grpc_client_tls("grpc.client", "grpc.master")?,
})
}
/// Connects to the cluster's storage and follows the masters' filer list.
/// Chunks are encrypted for the volume servers when the filer is
/// configured to (`weed filer -encryptVolumeData`), as `weed s3` does.
pub async fn storage(&self, security: &Security) -> Result<DirectReads> {
// The filers are dialed as the masters are: grpc.client's credentials
// and server allowlist, through this crate's own connector.
let filer_dialer: Option<FilerDialer> = security
.grpc_client_tls("grpc.client", "grpc.filer")?
.map(|tls| {
Arc::new(move |addr: &str| {
let channel = Endpoint::new(format!("http://{addr}"))?
.connect_timeout(Duration::from_secs(10))
.connect_with_connector_lazy(grpc_tls::connector(tls.clone()));
Ok(sw_client::FilerClient::from_channel(addr, channel))
}) as FilerDialer
});
let conn = Connection {
filers: discover_filers(&self.masters, &self.filer_group, self.master_tls.as_ref())
.await?,
read_jwt_key: security.jwt_read_key(),
sse: security.sse(),
grpc_tls: None,
https_tls: security.https_client_tls()?,
filer_dialer,
filer_group: self.filer_group.clone(),
};
let storage = DirectReads::connect(&conn)
.await
.context("direct storage")?;
tracing::info!(
"tables kept through {storage:?}; filer gRPC {}, volume servers {}; SSE-S3 objects {}",
if conn.filer_dialer.is_some() {
"over TLS"
} else {
"plaintext"
},
if conn.https_tls.is_some() {
"over HTTPS"
} else {
"over HTTP"
},
if storage.sse_s3_configured() {
"are readable"
} else {
"need s3.sse.kek or s3.sse.key"
},
);
tokio::spawn(follow_filers(
storage.clone(),
self.masters.clone(),
self.filer_group.clone(),
self.master_tls.clone(),
));
Ok(storage)
}
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn an_empty_master_list_is_refused() {
assert!(master_addresses("").is_err());
assert!(master_addresses(" , ").is_err());
assert_eq!(
master_addresses(" m1:9333,,m2:9333 ").unwrap(),
["m1:9333", "m2:9333"]
);
}
}
+76 -107
View File
@@ -1,9 +1,9 @@
//! Opening a Lance dataset with credentials the namespace vended.
//! Opening a Lance dataset on the cluster's storage.
//!
//! The worker holds no storage credentials of its own: it asks the namespace to
//! describe a table with `vend_credentials`, and the `storage_options` that come
//! back are handed to lance as-is. They expire, so a job that runs longer than
//! their lifetime re-describes rather than carrying one set throughout.
//! The namespace says where a table is; its files are read and written on the
//! filers and volume servers through [`DirectReads`]. The S3 store lance builds
//! for the location is unsigned and never dialed, so no process here holds S3
//! credentials. Tests on an in-process store open without storage.
use std::collections::HashMap;
use std::sync::Arc;
@@ -15,8 +15,10 @@ use lance::io::{
ObjectStore, ObjectStoreParams, ObjectStoreRegistry, StorageOptionsAccessor,
WrappingObjectStore,
};
use lance::session::Session;
use crate::catalog::{NamespaceClient, TableDescription};
use crate::direct_read::DirectReads;
/// A table the worker is about to work on.
pub struct OpenTable {
@@ -37,27 +39,43 @@ pub struct TableStats {
pub schema: Option<String>,
}
/// Storage options an operator supplies for deployments that vend none.
pub type FallbackOptions = HashMap<String, String>;
/// Where tables are kept: the cluster's filers and volume servers, or, for
/// tests on an in-process store, nowhere in particular.
#[derive(Clone, Default)]
pub struct Storage(pub Option<DirectReads>);
/// Resolve a table through the namespace and open it.
pub async fn open(
client: &NamespaceClient,
id: &[String],
fallback: &FallbackOptions,
) -> Result<OpenTable> {
open_with(client, id, fallback, None).await
impl Storage {
/// The store wrapper for a table at `location`, writing as the table's
/// bucket declares.
pub async fn wrapper(&self, location: &str) -> Result<Option<Arc<dyn WrappingObjectStore>>> {
let Some(direct) = &self.0 else {
return Ok(None);
};
let bucket = location
.strip_prefix("s3://")
.and_then(|rest| rest.split('/').next())
.with_context(|| format!("{location} names no bucket"))?;
let sse_s3 = direct.bucket_encrypts(bucket).await?;
Ok(Some(Arc::new(direct.encrypting(sse_s3))))
}
}
/// [`open`] with the dataset's object store wrapped, e.g. by a read cache.
pub async fn open_with(
client: &NamespaceClient,
id: &[String],
fallback: &FallbackOptions,
wrapper: Option<Arc<dyn WrappingObjectStore>>,
) -> Result<OpenTable> {
/// What lance builds its own S3 store with: unsigned, so it looks for no
/// credentials, and never dialed.
pub fn unsigned_options() -> HashMap<String, String> {
HashMap::from([
("aws_skip_signature".to_string(), "true".to_string()),
("aws_region".to_string(), "us-east-1".to_string()),
("aws_access_key_id".to_string(), String::new()),
("aws_secret_access_key".to_string(), String::new()),
])
}
/// Resolve a table through the namespace and open it.
pub async fn open(client: &NamespaceClient, id: &[String], storage: &Storage) -> Result<OpenTable> {
let description = client.describe_table(id).await?;
let dataset = open_at(&description, fallback, wrapper).await?;
let wrapper = storage.wrapper(&description.location).await?;
let dataset = open_at(&description, wrapper, None).await?;
Ok(OpenTable {
id: id.to_vec(),
location: description.location,
@@ -65,73 +83,57 @@ pub async fn open_with(
})
}
/// The object_store options for a table. The namespace vends object_store's own
/// option names, so they pass straight through. What it vends always wins: the
/// fallback exists because a deployment without STS vends no credentials at
/// all, and then the worker has no other way to reach the data.
pub fn table_options(
/// Open a table the caller has already resolved, with the store wrapped, in
/// `session` when given, otherwise in a session of its own.
pub async fn open_at(
description: &TableDescription,
fallback: &FallbackOptions,
) -> HashMap<String, String> {
let mut options: HashMap<String, String> = fallback.clone();
options.extend(description.storage_options.clone());
// A store that takes unsigned requests needs no credentials, but without
// any lance builds the AWS default chain, and that probes instance
// metadata on every open.
if options
.get("aws_skip_signature")
.is_some_and(|v| v == "true")
{
options.entry("aws_access_key_id".into()).or_default();
options.entry("aws_secret_access_key".into()).or_default();
}
options
}
async fn open_at(
description: &TableDescription,
fallback: &FallbackOptions,
wrapper: Option<Arc<dyn WrappingObjectStore>>,
session: Option<Arc<Session>>,
) -> Result<Dataset> {
DatasetBuilder::from_uri(&description.location)
.with_store_params(ObjectStoreParams {
storage_options_accessor: Some(Arc::new(StorageOptionsAccessor::with_static_options(
table_options(description, fallback),
))),
object_store_wrapper: wrapper,
..Default::default()
})
let mut builder =
DatasetBuilder::from_uri(&description.location).with_store_params(store_params(wrapper));
if let Some(session) = session {
builder = builder.with_session(session);
}
builder
.load()
.await
.with_context(|| format!("open lance dataset at {}", description.location))
}
/// The store parameters every open and write uses.
pub fn store_params(wrapper: Option<Arc<dyn WrappingObjectStore>>) -> ObjectStoreParams {
ObjectStoreParams {
storage_options_accessor: Some(Arc::new(StorageOptionsAccessor::with_static_options(
unsigned_options(),
))),
object_store_wrapper: wrapper,
..Default::default()
}
}
/// Remove every object under a table's location. A table the catalog can no
/// longer describe is already dropped, so the call stays idempotent.
pub async fn drop(
client: &NamespaceClient,
id: &[String],
fallback: &FallbackOptions,
) -> Result<()> {
pub async fn drop(client: &NamespaceClient, id: &[String], storage: &Storage) -> Result<()> {
let description = match client.describe_table(id).await {
Ok(description) => description,
Err(err) if describe_not_found(&err) => return Ok(()),
Err(err) => return Err(err),
};
check_drop_location(id, &description.location)?;
let (store, path) = ObjectStore::from_uri_and_params(
Arc::new(ObjectStoreRegistry::default()),
&description.location,
&ObjectStoreParams {
storage_options_accessor: Some(Arc::new(StorageOptionsAccessor::with_static_options(
table_options(&description, fallback),
))),
..Default::default()
},
)
.await?;
store.remove_dir_all(path).await?;
Ok(())
match &storage.0 {
Some(direct) => direct.remove_table(&description.location).await,
None => {
let (store, path) = ObjectStore::from_uri_and_params(
Arc::new(ObjectStoreRegistry::default()),
&description.location,
&store_params(None),
)
.await?;
store.remove_dir_all(path).await?;
Ok(())
}
}
}
/// Where the catalog puts a table it was given no location for, as the
@@ -218,45 +220,12 @@ fn schema_json(dataset: &Dataset) -> Option<String> {
#[cfg(test)]
mod tests {
use super::{FallbackOptions, check_drop_location, table_location, table_options};
use crate::catalog::TableDescription;
use super::{check_drop_location, table_location};
fn id(parts: &[&str]) -> Vec<String> {
parts.iter().map(|part| part.to_string()).collect()
}
fn options(pairs: &[(&str, &str)]) -> FallbackOptions {
pairs
.iter()
.map(|(k, v)| (k.to_string(), v.to_string()))
.collect()
}
#[test]
fn table_options_keep_an_unsigned_store_off_the_aws_default_chain() {
let described = |vended: &[(&str, &str)]| TableDescription {
location: "s3://bkt/vector/x".into(),
storage_options: options(vended),
managed_versioning: false,
};
let unsigned = described(&[("aws_skip_signature", "true")]);
let got = table_options(&unsigned, &FallbackOptions::new());
assert_eq!(got.get("aws_access_key_id").map(String::as_str), Some(""));
assert_eq!(
got.get("aws_secret_access_key").map(String::as_str),
Some("")
);
let fallback = options(&[("aws_access_key_id", "k"), ("aws_secret_access_key", "s")]);
let got = table_options(&unsigned, &fallback);
assert_eq!(got.get("aws_access_key_id").map(String::as_str), Some("k"));
let signed = described(&[("aws_endpoint", "http://s3:8333")]);
let got = table_options(&signed, &FallbackOptions::new());
assert!(!got.contains_key("aws_access_key_id"), "{got:?}");
}
#[test]
fn table_location_matches_the_gateway_layout() {
assert_eq!(
File diff suppressed because it is too large. Load diff
+948
View File
@@ -0,0 +1,948 @@
//! gRPC TLS the way weed processes do it.
//!
//! Serving: every client must present a certificate chaining to `grpc.ca`,
//! `allowed_commonNames` and `allowed_wildcard_domain` restrict which client
//! identities are served, and the certificate, key and CA files are re-read
//! on a timer and on SIGHUP so rotated files take effect without a restart.
//!
//! Dialing: the server's certificate must chain to `grpc.ca`; it is not
//! checked against the host dialed (weed clients verify the chain only, as
//! `advancedtls.CertVerification`), and `allowed_server_commonNames` /
//! `allowed_server_wildcard_domain` restrict which server identities are
//! accepted. The files are re-read at every dial.
use std::collections::HashSet;
use std::sync::{Arc, Mutex, RwLock};
use std::time::Duration;
use anyhow::{Context, Result, bail};
use futures::Stream;
use http::Uri;
use hyper_util::rt::TokioIo;
use rustls::client::danger::{HandshakeSignatureValid, ServerCertVerified, ServerCertVerifier};
use rustls::client::verify_server_cert_signed_by_trust_anchor;
use rustls::crypto::{CryptoProvider, aws_lc_rs};
use rustls::pki_types::pem::PemObject;
use rustls::pki_types::{CertificateDer, PrivateKeyDer, ServerName, UnixTime};
use rustls::server::danger::{ClientCertVerified, ClientCertVerifier};
use rustls::server::{ParsedCertificate, WebPkiClientVerifier};
use rustls::{
ClientConfig, DigitallySignedStruct, DistinguishedName, RootCertStore, ServerConfig,
SignatureScheme,
};
use tokio::net::{TcpListener, TcpStream};
use tokio_rustls::server::TlsStream;
use tokio_rustls::{TlsAcceptor, TlsConnector};
use x509_parser::extensions::GeneralName;
use x509_parser::prelude::{FromDer, X509Certificate};
use crate::security::{GrpcServerTls, GrpcTls};
/// `WEED_TLS_CERT_REFRESH_INTERVAL`, the cadence weed's PEM providers re-read
/// certificate files at; a Go duration such as `30m` or `500ms`.
pub const REFRESH_INTERVAL_ENV: &str = "WEED_TLS_CERT_REFRESH_INTERVAL";
const DEFAULT_REFRESH_INTERVAL: Duration = Duration::from_secs(5 * 3600);
const HANDSHAKE_TIMEOUT: Duration = Duration::from_secs(10);
/// How often the files are re-read: the environment's value, else weed's
/// default of five hours.
pub fn refresh_interval() -> Duration {
std::env::var(REFRESH_INTERVAL_ENV)
.ok()
.and_then(|s| parse_go_duration(&s))
.filter(|d| !d.is_zero())
.unwrap_or(DEFAULT_REFRESH_INTERVAL)
}
/// `time.ParseDuration`, with its arithmetic: an optional `+`, then a
/// sequence of `<number><unit>`, summed in whole nanoseconds (a fraction of
/// one is dropped) up to `i64::MAX`. A negative value is `None`, as weed
/// ignores one.
fn parse_go_duration(s: &str) -> Option<Duration> {
const LIMIT: u64 = 1 << 63;
let digits = |s: &str| s.find(|c: char| !c.is_ascii_digit()).unwrap_or(s.len());
let mut rest = s.strip_prefix('+').unwrap_or(s);
if rest == "0" {
return Some(Duration::ZERO);
}
if rest.is_empty() {
return None;
}
let mut total: u64 = 0;
while !rest.is_empty() {
let (whole, tail) = rest.split_at(digits(rest));
let mut v: u64 = 0;
for c in whole.bytes() {
if v > LIMIT / 10 {
return None;
}
v = v * 10 + u64::from(c - b'0');
if v > LIMIT {
return None;
}
}
let (mut fraction, mut scale) = (0u64, 1f64);
let (fraction_digits, tail) = match tail.strip_prefix('.') {
Some(tail) => tail.split_at(digits(tail)),
None => ("", tail),
};
let mut overflow = false;
for c in fraction_digits.bytes() {
if overflow || fraction > (LIMIT - 1) / 10 {
overflow = true;
continue;
}
let next = fraction * 10 + u64::from(c - b'0');
if next > LIMIT {
overflow = true;
continue;
}
fraction = next;
scale *= 10.0;
}
if whole.is_empty() && fraction_digits.is_empty() {
return None;
}
let unit_len = tail
.find(|c: char| c == '.' || c.is_ascii_digit())
.unwrap_or(tail.len());
let (unit, tail) = tail.split_at(unit_len);
let unit: u64 = match unit {
"ns" => 1,
"us" | "\u{b5}s" | "\u{3bc}s" => 1_000,
"ms" => 1_000_000,
"s" => 1_000_000_000,
"m" => 60_000_000_000,
"h" => 3_600_000_000_000,
_ => return None,
};
if v > LIMIT / unit {
return None;
}
v *= unit;
if fraction > 0 {
v += (fraction as f64 * (unit as f64 / scale)) as u64;
if v > LIMIT {
return None;
}
}
total = total.checked_add(v).filter(|&total| total <= LIMIT)?;
rest = tail;
}
(total < LIMIT).then(|| Duration::from_nanos(total))
}
/// The listener's TLS configuration, rebuilt from the files whenever they
/// change.
pub struct ServerTls {
files: GrpcServerTls,
config: RwLock<Arc<ServerConfig>>,
/// The file contents the current configuration was built from.
loaded: Mutex<Vec<u8>>,
}
impl std::fmt::Debug for ServerTls {
fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
f.debug_struct("ServerTls")
.field("files", &self.files)
.finish_non_exhaustive()
}
}
fn read(key: &str, path: &str) -> Result<Vec<u8>> {
std::fs::read(path).with_context(|| format!("{key} = {path:?}"))
}
fn read_all(files: &GrpcServerTls) -> Result<Vec<u8>> {
let mut all = Vec::new();
for (key, path) in [
("cert", &files.cert_path),
("key", &files.key_path),
("grpc.ca", &files.ca_path),
] {
all.extend(read(key, path)?);
all.push(0);
}
Ok(all)
}
fn cert_chain(key: &str, path: &str, pem: &[u8]) -> Result<Vec<CertificateDer<'static>>> {
let chain: Vec<CertificateDer<'static>> = CertificateDer::pem_slice_iter(pem)
.collect::<Result<_, _>>()
.with_context(|| format!("{key} = {path:?}"))?;
if chain.is_empty() {
bail!("{key} = {path:?} holds no certificate");
}
Ok(chain)
}
fn private_key(key: &str, path: &str, pem: &[u8]) -> Result<PrivateKeyDer<'static>> {
PrivateKeyDer::from_pem_slice(pem).with_context(|| format!("{key} = {path:?}"))
}
fn root_store(path: &str, pem: &[u8]) -> Result<RootCertStore> {
let mut roots = RootCertStore::empty();
for cert in cert_chain("grpc.ca", path, pem)? {
roots
.add(cert)
.with_context(|| format!("grpc.ca = {path:?}"))?;
}
Ok(roots)
}
/// Builds the configuration `security.LoadServerTLS` would: the identity in
/// `cert`/`key`, every client verified against `grpc.ca`, and the client's
/// common name checked against the allowlist when one is set.
fn build_config(files: &GrpcServerTls) -> Result<ServerConfig> {
let cert_chain = cert_chain("cert", &files.cert_path, &read("cert", &files.cert_path)?)?;
let key = private_key("key", &files.key_path, &read("key", &files.key_path)?)?;
let roots = root_store(&files.ca_path, &read("grpc.ca", &files.ca_path)?)?;
let verifier = WebPkiClientVerifier::builder(Arc::new(roots))
.build()
.context("client certificate verifier")?;
let verifier: Arc<dyn ClientCertVerifier> =
if files.allowed_common_names.is_empty() && files.allowed_wildcard_domain.is_empty() {
verifier
} else {
Arc::new(CommonNameVerifier {
inner: verifier,
allowed_common_names: files.allowed_common_names.iter().cloned().collect(),
allowed_wildcard_domain: files.allowed_wildcard_domain.clone(),
})
};
let mut config = ServerConfig::builder()
.with_client_cert_verifier(verifier)
.with_single_cert(cert_chain, key)
.context("server certificate")?;
config.alpn_protocols = vec![b"h2".to_vec()];
Ok(config)
}
impl ServerTls {
pub fn load(files: GrpcServerTls) -> Result<Arc<Self>> {
crate::tls::install_default_crypto_provider();
let loaded = read_all(&files)?;
let config = build_config(&files)?;
Ok(Arc::new(Self {
files,
config: RwLock::new(Arc::new(config)),
loaded: Mutex::new(loaded),
}))
}
pub fn files(&self) -> &GrpcServerTls {
&self.files
}
/// Re-reads the files; when they changed, the next handshake uses them.
/// Returns whether they had. A file that no longer parses is an error,
/// and the configuration in use stays.
pub fn reload(&self) -> Result<bool> {
let current = read_all(&self.files)?;
let mut loaded = self.loaded.lock().unwrap_or_else(|e| e.into_inner());
if *loaded == current {
return Ok(false);
}
let config = build_config(&self.files)?;
*self.config.write().unwrap_or_else(|e| e.into_inner()) = Arc::new(config);
*loaded = current;
Ok(true)
}
pub fn acceptor(&self) -> TlsAcceptor {
TlsAcceptor::from(
self.config
.read()
.unwrap_or_else(|e| e.into_inner())
.clone(),
)
}
/// Connections accepted on `listener` and taken through the TLS
/// handshake with the configuration current at that moment. Each
/// handshake runs on its own task under `HANDSHAKE_TIMEOUT`, so a peer
/// that connects and stays silent holds up nobody else; a failed
/// handshake is logged and skipped, as the Rust volume server does.
pub fn incoming(
self: &Arc<Self>,
listener: TcpListener,
) -> impl Stream<Item = std::io::Result<TlsStream<TcpStream>>> + Send + 'static {
let tls = self.clone();
let (tx, rx) = tokio::sync::mpsc::channel(64);
tokio::spawn(async move {
loop {
let accepted = tokio::select! {
_ = tx.closed() => return,
accepted = listener.accept() => accepted,
};
let (stream, peer) = match accepted {
Ok(accepted) => accepted,
Err(err) => {
let _ = tx.send(Err(err)).await;
continue;
}
};
let _ = stream.set_nodelay(true);
let acceptor = tls.acceptor();
let tx = tx.clone();
tokio::spawn(async move {
match tokio::time::timeout(HANDSHAKE_TIMEOUT, acceptor.accept(stream)).await {
Ok(Ok(stream)) => {
let _ = tx.send(Ok(stream)).await;
}
Ok(Err(err)) => tracing::debug!("gRPC TLS handshake from {peer}: {err}"),
Err(_) => tracing::debug!("gRPC TLS handshake from {peer} timed out"),
}
});
}
});
tokio_stream::wrappers::ReceiverStream::new(rx)
}
/// Re-reads the files every `every`, and at once on SIGHUP.
pub async fn refresh(self: Arc<Self>, every: Duration) {
#[cfg(unix)]
let mut hangup =
tokio::signal::unix::signal(tokio::signal::unix::SignalKind::hangup()).ok();
loop {
#[cfg(unix)]
{
tokio::select! {
_ = tokio::time::sleep(every) => {}
_ = async { match hangup.as_mut() { Some(h) => h.recv().await, None => std::future::pending().await } } => {
tracing::info!("SIGHUP: re-reading gRPC TLS files");
}
}
}
#[cfg(not(unix))]
tokio::time::sleep(every).await;
match self.reload() {
Ok(true) => tracing::info!("gRPC TLS files changed; new handshakes use them"),
Ok(false) => {}
Err(err) => tracing::warn!("gRPC TLS files not reloaded: {err:#}"),
}
}
}
}
/// `security.Authenticator` for client certificates: after the chain is
/// verified, the subject common name must be allowed, or lie under the
/// wildcard domain.
#[derive(Debug)]
struct CommonNameVerifier {
inner: Arc<dyn ClientCertVerifier>,
allowed_common_names: HashSet<String>,
allowed_wildcard_domain: String,
}
impl ClientCertVerifier for CommonNameVerifier {
fn offer_client_auth(&self) -> bool {
self.inner.offer_client_auth()
}
fn client_auth_mandatory(&self) -> bool {
self.inner.client_auth_mandatory()
}
fn root_hint_subjects(&self) -> &[DistinguishedName] {
self.inner.root_hint_subjects()
}
fn verify_client_cert(
&self,
end_entity: &CertificateDer<'_>,
intermediates: &[CertificateDer<'_>],
now: UnixTime,
) -> Result<ClientCertVerified, rustls::Error> {
self.inner
.verify_client_cert(end_entity, intermediates, now)?;
let common_name = parse_common_name(end_entity).map_err(|e| {
rustls::Error::General(format!("parse client certificate common name failed: {e}"))
})?;
if common_name_is_allowed(
&common_name,
&self.allowed_common_names,
&self.allowed_wildcard_domain,
) {
return Ok(ClientCertVerified::assertion());
}
Err(rustls::Error::General(format!(
"Authenticate: invalid subject client common name: {common_name}"
)))
}
fn verify_tls12_signature(
&self,
message: &[u8],
cert: &CertificateDer<'_>,
dss: &DigitallySignedStruct,
) -> Result<HandshakeSignatureValid, rustls::Error> {
self.inner.verify_tls12_signature(message, cert, dss)
}
fn verify_tls13_signature(
&self,
message: &[u8],
cert: &CertificateDer<'_>,
dss: &DigitallySignedStruct,
) -> Result<HandshakeSignatureValid, rustls::Error> {
self.inner.verify_tls13_signature(message, cert, dss)
}
fn supported_verify_schemes(&self) -> Vec<SignatureScheme> {
self.inner.supported_verify_schemes()
}
}
/// The client configuration `security.LoadClientTLS` would build, from the
/// CA, certificate and key as `pems` holds them.
fn client_config(
files: &GrpcTls,
(ca, cert, key): (Vec<u8>, Vec<u8>, Vec<u8>),
) -> Result<ClientConfig> {
crate::tls::install_default_crypto_provider();
let verifier = ServerIdentityVerifier {
roots: Arc::new(root_store(&files.ca_path, &ca)?),
provider: Arc::new(aws_lc_rs::default_provider()),
allowed_names: files.allowed_server_names.iter().cloned().collect(),
allowed_wildcard_domain: files.allowed_server_wildcard_domain.clone(),
};
let mut config = ClientConfig::builder()
.dangerous()
.with_custom_certificate_verifier(Arc::new(verifier))
.with_client_auth_cert(
cert_chain("cert", &files.cert_path, &cert)?,
private_key("key", &files.key_path, &key)?,
)
.context("client certificate")?;
config.alpn_protocols = vec![b"h2".to_vec()];
Ok(config)
}
/// The host of a gRPC address, without an IPv6 address's brackets.
fn dialed_host(uri: &Uri) -> Option<&str> {
Some(uri.host()?.trim_matches(|c| c == '[' || c == ']'))
}
/// A tonic connector for `Endpoint::connect_with_connector`: the files read
/// as they are now, so rotated credentials are picked up, then TCP to the
/// URI's host and port and the handshake under [`client_config`].
pub fn connector(
files: GrpcTls,
) -> impl tower::Service<
Uri,
Response = TokioIo<tokio_rustls::client::TlsStream<TcpStream>>,
Error = Box<dyn std::error::Error + Send + Sync>,
Future = impl Send,
> + Clone
+ Send
+ 'static {
tower::service_fn(move |uri: Uri| {
let files = files.clone();
async move {
let host = dialed_host(&uri).ok_or("no host in gRPC address")?;
let port = uri.port_u16().unwrap_or(443);
let config = Arc::new(client_config(&files, files.pems().await?)?);
let stream = TcpStream::connect((host, port)).await?;
let _ = stream.set_nodelay(true);
let name = ServerName::try_from(host.to_string())?;
let stream = TlsConnector::from(config).connect(name, stream).await?;
Ok(TokioIo::new(stream))
}
})
}
/// `security.Authenticator.AuthenticateServer`: the chain must reach
/// `grpc.ca`, the host dialed is not checked, and when an allowlist is set the
/// common name or a DNS SAN must be allowed or lie under the wildcard domain,
/// while a URI SAN must be allowed exactly.
#[derive(Debug)]
struct ServerIdentityVerifier {
roots: Arc<RootCertStore>,
provider: Arc<CryptoProvider>,
allowed_names: HashSet<String>,
allowed_wildcard_domain: String,
}
impl ServerCertVerifier for ServerIdentityVerifier {
fn verify_server_cert(
&self,
end_entity: &CertificateDer<'_>,
intermediates: &[CertificateDer<'_>],
_server_name: &ServerName<'_>,
_ocsp_response: &[u8],
now: UnixTime,
) -> Result<ServerCertVerified, rustls::Error> {
let cert = ParsedCertificate::try_from(end_entity)?;
verify_server_cert_signed_by_trust_anchor(
&cert,
&self.roots,
intermediates,
now,
self.provider.signature_verification_algorithms.all,
)?;
if self.allowed_names.is_empty() && self.allowed_wildcard_domain.is_empty() {
return Ok(ServerCertVerified::assertion());
}
let (names, uris) = server_identities(end_entity)
.map_err(|e| rustls::Error::General(format!("parse server certificate failed: {e}")))?;
if names.iter().any(|name| {
self.allowed_names.contains(name)
|| wildcard_domain_matches(name, &self.allowed_wildcard_domain)
}) || uris.iter().any(|uri| self.allowed_names.contains(uri))
{
return Ok(ServerCertVerified::assertion());
}
Err(rustls::Error::General(format!(
"AuthenticateServer: untrusted server identity {:?}",
names.into_iter().chain(uris).collect::<Vec<_>>()
)))
}
fn verify_tls12_signature(
&self,
message: &[u8],
cert: &CertificateDer<'_>,
dss: &DigitallySignedStruct,
) -> Result<HandshakeSignatureValid, rustls::Error> {
rustls::crypto::verify_tls12_signature(
message,
cert,
dss,
&self.provider.signature_verification_algorithms,
)
}
fn verify_tls13_signature(
&self,
message: &[u8],
cert: &CertificateDer<'_>,
dss: &DigitallySignedStruct,
) -> Result<HandshakeSignatureValid, rustls::Error> {
rustls::crypto::verify_tls13_signature(
message,
cert,
dss,
&self.provider.signature_verification_algorithms,
)
}
fn supported_verify_schemes(&self) -> Vec<SignatureScheme> {
self.provider
.signature_verification_algorithms
.supported_schemes()
}
}
/// The common name and DNS SANs, and the URI SANs, of a certificate.
fn server_identities(cert: &CertificateDer<'_>) -> Result<(Vec<String>, Vec<String>)> {
let (_, certificate) = X509Certificate::from_der(cert.as_ref())
.map_err(|e| anyhow::anyhow!("parse X.509 certificate failed: {e}"))?;
let mut names = vec![
certificate
.subject()
.iter_common_name()
.next()
.and_then(|common_name| common_name.as_str().ok())
.unwrap_or_default()
.to_string(),
];
let mut uris = Vec::new();
if let Ok(Some(san)) = certificate.subject_alternative_name() {
for name in &san.value.general_names {
match name {
GeneralName::DNSName(dns) => names.push(dns.to_string()),
GeneralName::URI(uri) => uris.push(uri.to_string()),
_ => {}
}
}
}
Ok((names, uris))
}
fn parse_common_name(cert: &CertificateDer<'_>) -> Result<String> {
let (_, certificate) = X509Certificate::from_der(cert.as_ref())
.map_err(|e| anyhow::anyhow!("parse X.509 certificate failed: {e}"))?;
Ok(certificate
.subject()
.iter_common_name()
.next()
.and_then(|common_name| common_name.as_str().ok())
.map(str::to_string)
.unwrap_or_default())
}
fn common_name_is_allowed(
common_name: &str,
allowed_common_names: &HashSet<String>,
allowed_wildcard_domain: &str,
) -> bool {
allowed_common_names.contains(common_name)
|| wildcard_domain_matches(common_name, allowed_wildcard_domain)
}
// Mirrors the Go wildcard check: the name must end with the domain, carry a
// non-empty label in front of it, and be a valid DNS name so a CA-signed name
// like ".example.com" cannot satisfy the suffix check. A leading "*." label
// is allowed so wildcard certs keep matching. One trailing root dot is
// ignored on both sides so absolute DNS names still match.
fn wildcard_domain_matches(name: &str, wildcard_domain: &str) -> bool {
let name = name.strip_suffix('.').unwrap_or(name);
let wildcard_domain = wildcard_domain.strip_suffix('.').unwrap_or(wildcard_domain);
if wildcard_domain.is_empty()
|| name.len() <= wildcard_domain.len()
|| !name.ends_with(wildcard_domain)
{
return false;
}
if !wildcard_domain.starts_with('.')
&& name.as_bytes()[name.len() - wildcard_domain.len() - 1] != b'.'
{
return false;
}
is_domain_name(name.strip_prefix("*.").unwrap_or(name))
}
// Mirrors Go's net.IsDomainName: non-empty labels of letters, digits, '-',
// '_', each up to 63 chars and not '-'-edged, with at least one non-digit so
// a bare IP cannot pass a domain wildcard.
fn is_domain_name(name: &str) -> bool {
if name.is_empty() || name.len() > 253 {
return false;
}
let mut non_numeric = false;
let valid = name.split('.').all(|label| {
non_numeric |= label.bytes().any(|c| !c.is_ascii_digit());
!label.is_empty()
&& label.len() <= 63
&& !label.starts_with('-')
&& !label.ends_with('-')
&& label
.bytes()
.all(|c| c.is_ascii_alphanumeric() || c == b'-' || c == b'_')
});
valid && non_numeric
}
#[cfg(test)]
mod tests {
use super::*;
use std::path::{Path, PathBuf};
/// `tests/tls`: a CA and two leaves it signed, `engine-a` and `engine-b`.
fn fixture(name: &str) -> PathBuf {
Path::new(env!("CARGO_MANIFEST_DIR"))
.join("tests/tls")
.join(name)
}
fn install(dir: &Path, leaf: &str) -> GrpcServerTls {
for (from, to) in [
(format!("{leaf}.pem"), "cert.pem"),
(format!("{leaf}.key"), "key.pem"),
("ca.pem".to_string(), "ca.pem"),
] {
std::fs::copy(fixture(&from), dir.join(to)).unwrap();
}
GrpcServerTls {
cert_path: dir.join("cert.pem").display().to_string(),
key_path: dir.join("key.pem").display().to_string(),
ca_path: dir.join("ca.pem").display().to_string(),
allowed_common_names: Vec::new(),
allowed_wildcard_domain: String::new(),
}
}
#[test]
fn rotated_files_are_picked_up_and_broken_ones_are_not() {
let dir = tempfile::tempdir().unwrap();
let tls = ServerTls::load(install(dir.path(), "a")).unwrap();
assert!(!tls.reload().unwrap(), "unchanged files are not a reload");
let before = tls.acceptor();
install(dir.path(), "b");
assert!(tls.reload().unwrap());
let after = tls.acceptor();
assert!(!Arc::ptr_eq(before.config(), after.config()));
std::fs::write(dir.path().join("key.pem"), "not a key").unwrap();
assert!(tls.reload().is_err());
assert!(
Arc::ptr_eq(after.config(), tls.acceptor().config()),
"a broken file leaves the working configuration in place"
);
}
#[tokio::test]
async fn a_silent_peer_does_not_hold_up_the_next_handshake() {
use futures::StreamExt;
crate::tls::install_default_crypto_provider();
let dir = tempfile::tempdir().unwrap();
let tls = ServerTls::load(install(dir.path(), "a")).unwrap();
let listener = TcpListener::bind("127.0.0.1:0").await.unwrap();
let addr = listener.local_addr().unwrap();
let mut incoming = Box::pin(tls.incoming(listener));
let _silent = TcpStream::connect(addr).await.unwrap();
let mut roots = RootCertStore::empty();
for cert in CertificateDer::pem_file_iter(fixture("ca.pem")).unwrap() {
roots.add(cert.unwrap()).unwrap();
}
let config = rustls::ClientConfig::builder()
.with_root_certificates(roots)
.with_client_auth_cert(
vec![CertificateDer::from_pem_file(fixture("a.pem")).unwrap()],
PrivateKeyDer::from_pem_file(fixture("a.key")).unwrap(),
)
.unwrap();
let client = tokio::spawn(async move {
let tcp = TcpStream::connect(addr).await.unwrap();
tokio_rustls::TlsConnector::from(Arc::new(config))
.connect("localhost".try_into().unwrap(), tcp)
.await
});
let served = tokio::time::timeout(Duration::from_secs(5), incoming.next()).await;
assert!(
matches!(served, Ok(Some(Ok(_)))),
"the handshake waited behind a silent peer"
);
client.await.unwrap().unwrap();
}
#[test]
fn missing_or_empty_files_are_refused() {
let dir = tempfile::tempdir().unwrap();
let mut files = install(dir.path(), "a");
files.ca_path = dir.path().join("absent.pem").display().to_string();
assert!(ServerTls::load(files.clone()).is_err());
std::fs::write(dir.path().join("empty.pem"), "").unwrap();
files.ca_path = dir.path().join("empty.pem").display().to_string();
assert!(ServerTls::load(files).is_err());
}
#[test]
fn client_common_names_follow_the_allowlist() {
crate::tls::install_default_crypto_provider();
let mut roots = RootCertStore::empty();
for cert in CertificateDer::pem_file_iter(fixture("ca.pem")).unwrap() {
roots.add(cert.unwrap()).unwrap();
}
let inner = WebPkiClientVerifier::builder(Arc::new(roots))
.build()
.unwrap();
let leaf = |name: &str| CertificateDer::from_pem_file(fixture(name)).unwrap();
let now = UnixTime::now();
// Without an allowlist any certificate the CA signed is served.
assert!(inner.verify_client_cert(&leaf("a.pem"), &[], now).is_ok());
assert!(inner.verify_client_cert(&leaf("b.pem"), &[], now).is_ok());
let verifier = CommonNameVerifier {
inner,
allowed_common_names: ["engine-a".to_string()].into_iter().collect(),
allowed_wildcard_domain: String::new(),
};
assert!(
verifier
.verify_client_cert(&leaf("a.pem"), &[], now)
.is_ok()
);
let err = verifier
.verify_client_cert(&leaf("b.pem"), &[], now)
.unwrap_err();
assert!(err.to_string().contains("engine-b"), "{err}");
assert_eq!(parse_common_name(&leaf("b.pem")).unwrap(), "engine-b");
}
#[test]
fn server_identities_follow_the_allowlist() {
crate::tls::install_default_crypto_provider();
let leaf = |name: &str| CertificateDer::from_pem_file(fixture(name)).unwrap();
let verifier = |allowed: &[&str], wildcard: &str| ServerIdentityVerifier {
roots: Arc::new(
root_store("ca.pem", &std::fs::read(fixture("ca.pem")).unwrap()).unwrap(),
),
provider: Arc::new(aws_lc_rs::default_provider()),
allowed_names: allowed.iter().map(|s| s.to_string()).collect(),
allowed_wildcard_domain: wildcard.into(),
};
let name = ServerName::try_from("nothing.example").unwrap();
let check = |v: &ServerIdentityVerifier, cert: &str| {
v.verify_server_cert(&leaf(cert), &[], &name, &[], UnixTime::now())
};
// The chain is checked; the host dialed is not.
assert!(check(&verifier(&[], ""), "a.pem").is_ok());
assert!(check(&verifier(&[], ""), "b.pem").is_ok());
assert!(
check(&verifier(&[], ""), "ca.pem").is_err(),
"a root is not a server"
);
// The common name or a DNS SAN must be allowed.
assert!(check(&verifier(&["engine-a"], ""), "a.pem").is_ok());
assert!(check(&verifier(&["localhost"], ""), "b.pem").is_ok());
let err = check(&verifier(&["engine-a"], ""), "b.pem").unwrap_err();
assert!(err.to_string().contains("engine-b"), "{err}");
assert!(check(&verifier(&[], ".example"), "b.pem").is_err());
assert_eq!(
server_identities(&leaf("a.pem")).unwrap(),
(
vec!["engine-a".to_string(), "localhost".to_string()],
vec![]
)
);
}
#[test]
fn the_dialed_host_drops_ipv6_brackets() {
for (addr, host) in [
("http://m1:19333", "m1"),
("http://master.example", "master.example"),
("http://127.0.0.1:19333", "127.0.0.1"),
("http://[::1]:19333", "::1"),
] {
let uri: Uri = addr.parse().unwrap();
assert_eq!(dialed_host(&uri), Some(host), "{addr}");
}
}
fn client(leaf: &str, allowed: &[&str]) -> GrpcTls {
GrpcTls {
ca_path: fixture("ca.pem").display().to_string(),
cert_path: fixture(&format!("{leaf}.pem")).display().to_string(),
key_path: fixture(&format!("{leaf}.key")).display().to_string(),
allowed_server_names: allowed.iter().map(|s| s.to_string()).collect(),
allowed_server_wildcard_domain: String::new(),
}
}
async fn handshake(port: u16, files: &GrpcTls) -> Result<()> {
let config = Arc::new(client_config(files, files.pems().await?)?);
let stream = TcpStream::connect(("127.0.0.1", port)).await?;
let name = ServerName::try_from("127.0.0.1").unwrap();
let mut stream = TlsConnector::from(config).connect(name, stream).await?;
// A server that rejected the client certificate says so after the
// handshake, on the first read.
let mut byte = [0u8; 1];
match tokio::time::timeout(
Duration::from_secs(1),
tokio::io::AsyncReadExt::read(&mut stream, &mut byte),
)
.await
{
Ok(Err(err)) => Err(err.into()),
_ => Ok(()),
}
}
#[tokio::test]
async fn both_sides_check_identities() {
use futures::StreamExt;
let dir = tempfile::tempdir().unwrap();
let mut files = install(dir.path(), "a");
files.allowed_common_names = vec!["engine-b".into()];
let tls = ServerTls::load(files).unwrap();
let listener = TcpListener::bind("127.0.0.1:0").await.unwrap();
let port = listener.local_addr().unwrap().port();
let mut incoming = Box::pin(tls.incoming(listener));
// engine-b, trusting the CA, accepts engine-a by name, dialed by IP.
let allowed = client("b", &["engine-a"]);
let (served, dialed) = tokio::join!(
tokio::time::timeout(Duration::from_secs(5), incoming.next()),
handshake(port, &allowed)
);
assert!(dialed.is_ok(), "{dialed:?}");
assert!(matches!(served, Ok(Some(Ok(_)))));
// The client refuses a server outside its allowlist.
let err = handshake(port, &client("b", &["engine-z"]))
.await
.unwrap_err();
assert!(
err.to_string().contains("untrusted server identity"),
"{err}"
);
// The server refuses a client outside its allowlist: engine-a is not
// engine-b.
let outsider = client("a", &[]);
let (served, dialed) = tokio::join!(
tokio::time::timeout(Duration::from_secs(1), incoming.next()),
handshake(port, &outsider)
);
assert!(
dialed.is_err(),
"the server must cut off a client it does not allow"
);
assert!(served.is_err(), "nothing is served for a refused client");
}
#[test]
fn wildcard_domain_matches_like_go() {
let none = HashSet::new();
assert!(common_name_is_allowed("a.internal", &none, ".internal"));
assert!(common_name_is_allowed("a.internal", &none, "internal"));
assert!(common_name_is_allowed("*.internal", &none, ".internal"));
assert!(!common_name_is_allowed("internal", &none, ".internal"));
assert!(!common_name_is_allowed("xinternal", &none, "internal"));
assert!(!common_name_is_allowed("10.0.0.1", &none, "0.0.1"));
assert!(!common_name_is_allowed("a.internal", &none, ""));
let names: HashSet<String> = ["gateway".to_string()].into_iter().collect();
assert!(common_name_is_allowed("gateway", &names, ""));
assert!(!common_name_is_allowed("other", &names, ""));
}
#[test]
fn refresh_interval_reads_go_durations() {
assert_eq!(parse_go_duration("30m"), Some(Duration::from_secs(1800)));
assert_eq!(parse_go_duration("500ms"), Some(Duration::from_millis(500)));
assert_eq!(parse_go_duration("1h30m"), Some(Duration::from_secs(5400)));
assert_eq!(parse_go_duration("2.5s"), Some(Duration::from_millis(2500)));
assert_eq!(parse_go_duration("+2s"), Some(Duration::from_secs(2)));
assert_eq!(
parse_go_duration("1500us"),
Some(Duration::from_micros(1500))
);
assert_eq!(
parse_go_duration("3\u{b5}s"),
Some(Duration::from_micros(3))
);
assert_eq!(
parse_go_duration("3\u{3bc}s"),
Some(Duration::from_micros(3))
);
assert_eq!(parse_go_duration("1s20ns"), Some(Duration::new(1, 20)));
assert_eq!(parse_go_duration("-5m"), None);
assert_eq!(parse_go_duration("99999999999999999999h"), None);
assert_eq!(
parse_go_duration("4000000000000000h4000000000000000h"),
None
);
assert_eq!(parse_go_duration("2d"), None);
assert_eq!(parse_go_duration(""), None);
assert_eq!(parse_go_duration("0"), Some(Duration::ZERO));
assert_eq!(parse_go_duration("0.3s"), Some(Duration::from_millis(300)));
assert_eq!(parse_go_duration("1.1s"), Some(Duration::from_millis(1100)));
assert_eq!(parse_go_duration(".5h"), Some(Duration::from_secs(1800)));
assert_eq!(parse_go_duration("5.s"), Some(Duration::from_secs(5)));
assert_eq!(parse_go_duration(".s"), None);
assert_eq!(parse_go_duration("1.2.3s"), None);
assert_eq!(parse_go_duration(" 30m"), None);
// A fraction of a nanosecond is dropped, so this is zero and the
// default applies, as in weed.
assert_eq!(parse_go_duration("0.9ns"), Some(Duration::ZERO));
assert_eq!(parse_go_duration("1.9ns"), Some(Duration::from_nanos(1)));
// The longest time.Duration, and past it.
assert_eq!(
parse_go_duration("2562047h47m16.854775807s"),
Some(Duration::from_nanos(i64::MAX as u64))
);
assert_eq!(
parse_go_duration("9223372036854775807ns"),
Some(Duration::from_nanos(i64::MAX as u64))
);
assert_eq!(parse_go_duration("9223372036854775808ns"), None);
assert_eq!(parse_go_duration("2562048h"), None);
assert_eq!(parse_go_duration("3000000h"), None);
}
}
@@ -31,7 +31,7 @@ const MAX_MIN_VERSIONS: i64 = 1000;
/// the cluster owned the policy than each writer.
pub struct CleanupVersionsHandler {
namespace_url: String,
fallback: dataset::FallbackOptions,
storage: dataset::Storage,
signing_key: Option<SigningKey>,
metrics: Option<crate::metrics::LanceMetrics>,
}
@@ -40,7 +40,7 @@ impl CleanupVersionsHandler {
pub fn new(namespace_url: String) -> Self {
Self {
namespace_url,
fallback: dataset::FallbackOptions::new(),
storage: dataset::Storage::default(),
signing_key: None,
metrics: None,
}
@@ -51,8 +51,8 @@ impl CleanupVersionsHandler {
self
}
pub fn with_fallback(mut self, fallback: dataset::FallbackOptions) -> Self {
self.fallback = fallback;
pub fn with_storage(mut self, storage: dataset::Storage) -> Self {
self.storage = storage;
self
}
@@ -62,9 +62,7 @@ impl CleanupVersionsHandler {
}
fn client(&self) -> NamespaceClient {
NamespaceClient::new(self.namespace_url.clone())
.with_fallback(&self.fallback)
.with_signing_key(self.signing_key.as_ref())
NamespaceClient::new(self.namespace_url.clone()).with_signing_key(self.signing_key.as_ref())
}
}
@@ -160,7 +158,7 @@ impl JobHandler for CleanupVersionsHandler {
let mut proposals = Vec::new();
for encoded in &tables {
let id = parse_id(encoded);
let table = match dataset::open(&client, &id, &self.fallback).await {
let table = match dataset::open(&client, &id, &self.storage).await {
Ok(table) => table,
Err(err) => {
if let Some(counters) = &self.metrics {
@@ -256,7 +254,7 @@ impl JobHandler for CleanupVersionsHandler {
) as usize;
let client = self.client();
let table = dataset::open(&client, &id, &self.fallback).await?;
let table = dataset::open(&client, &id, &self.storage).await?;
let before = table.stats().await?;
// The floor is a promise about how much history survives, so it has to
@@ -34,7 +34,7 @@ const MIN_FRAGMENTS_CEILING: i64 = 4096;
/// accumulates small files the same way an Iceberg table does.
pub struct CompactHandler {
namespace_url: String,
fallback: dataset::FallbackOptions,
storage: dataset::Storage,
signing_key: Option<SigningKey>,
metrics: Option<crate::metrics::LanceMetrics>,
}
@@ -43,7 +43,7 @@ impl CompactHandler {
pub fn new(namespace_url: String) -> Self {
Self {
namespace_url,
fallback: dataset::FallbackOptions::new(),
storage: dataset::Storage::default(),
signing_key: None,
metrics: None,
}
@@ -55,8 +55,8 @@ impl CompactHandler {
}
/// Storage options to use where the namespace vends none.
pub fn with_fallback(mut self, fallback: dataset::FallbackOptions) -> Self {
self.fallback = fallback;
pub fn with_storage(mut self, storage: dataset::Storage) -> Self {
self.storage = storage;
self
}
@@ -66,9 +66,7 @@ impl CompactHandler {
}
fn client(&self) -> NamespaceClient {
NamespaceClient::new(self.namespace_url.clone())
.with_fallback(&self.fallback)
.with_signing_key(self.signing_key.as_ref())
NamespaceClient::new(self.namespace_url.clone()).with_signing_key(self.signing_key.as_ref())
}
}
@@ -154,7 +152,7 @@ impl JobHandler for CompactHandler {
let mut observations = Vec::new();
for encoded in &tables {
let id = parse_id(encoded);
let table = match dataset::open(&client, &id, &self.fallback).await {
let table = match dataset::open(&client, &id, &self.storage).await {
Ok(table) => table,
Err(err) => {
// A table that cannot be opened is the next run's problem,
@@ -271,7 +269,7 @@ impl JobHandler for CompactHandler {
let client = self.client();
// Re-resolve rather than trusting the location detection saw: the table
// may have been repointed, and the vended credentials have expired.
let mut table = dataset::open(&client, &id, &self.fallback).await?;
let mut table = dataset::open(&client, &id, &self.storage).await?;
let before = table.stats().await?;
sender.send_progress(JobProgressUpdate {
+131 -34
View File
@@ -6,8 +6,8 @@ use lance::index::DatasetIndexExt;
use lance_index::optimize::OptimizeOptions;
use seaweed_worker_core::config_form::{form, int_or, int_value, number_field};
use seaweed_worker_core::pb::{
ConfigValue, DetectionComplete, DetectionProposals, ExecuteJobRequest, JobCompleted,
JobProgressUpdate, JobProposal, JobResult, JobTypeCapability, JobTypeDescriptor,
AdminRuntimeDefaults, ConfigValue, DetectionComplete, DetectionProposals, ExecuteJobRequest,
JobCompleted, JobProgressUpdate, JobProposal, JobResult, JobTypeCapability, JobTypeDescriptor,
RunDetectionRequest,
};
use seaweed_worker_core::{DetectionSender, ExecutionSender, JobHandler};
@@ -15,9 +15,13 @@ use tracing::warn;
use crate::catalog::{NamespaceClient, SigningKey, parse_id};
use crate::dataset::{self, OpenTable};
use crate::jobs::vector_index::is_vector_table;
use crate::jobs::{clamp, string_list, table_id};
use crate::vector_index;
pub const JOB_TYPE: &str = "lance_optimize_indices";
/// The vector column the S3 Vectors API writes.
const VECTOR_COLUMN: &str = "vector";
const DEFAULT_MAX_UNINDEXED_ROWS: i64 = 100_000;
@@ -34,7 +38,7 @@ const MAX_UNINDEXED_CEILING: i64 = 100_000_000;
/// slow one.
pub struct OptimizeIndicesHandler {
namespace_url: String,
fallback: dataset::FallbackOptions,
storage: dataset::Storage,
signing_key: Option<SigningKey>,
metrics: Option<crate::metrics::LanceMetrics>,
}
@@ -43,7 +47,7 @@ impl OptimizeIndicesHandler {
pub fn new(namespace_url: String) -> Self {
Self {
namespace_url,
fallback: dataset::FallbackOptions::new(),
storage: dataset::Storage::default(),
signing_key: None,
metrics: None,
}
@@ -54,8 +58,8 @@ impl OptimizeIndicesHandler {
self
}
pub fn with_fallback(mut self, fallback: dataset::FallbackOptions) -> Self {
self.fallback = fallback;
pub fn with_storage(mut self, storage: dataset::Storage) -> Self {
self.storage = storage;
self
}
@@ -65,22 +69,38 @@ impl OptimizeIndicesHandler {
}
fn client(&self) -> NamespaceClient {
NamespaceClient::new(self.namespace_url.clone())
.with_fallback(&self.fallback)
.with_signing_key(self.signing_key.as_ref())
NamespaceClient::new(self.namespace_url.clone()).with_signing_key(self.signing_key.as_ref())
}
}
/// Rows no index covers, summed across a table's indices. A table with no
/// indices at all has nothing to optimize, which is different from a table whose
/// indices have fallen behind.
async fn unindexed_rows(table: &OpenTable) -> Result<Option<u64>> {
/// Rows the table's indices cover and rows they miss. A vector table's indices
/// are taken together, since a sharded build gives one name per shard; any
/// other table reports the index that misses the most. A table with no
/// indices at all has nothing to optimize, which is different from a table
/// whose indices have fallen behind.
struct IndexRows {
indexed: u64,
unindexed: u64,
}
async fn index_rows(table: &OpenTable, vector_table: bool) -> Result<Option<IndexRows>> {
if vector_table {
return Ok(vector_index::coverage(&table.dataset, VECTOR_COLUMN)
.await?
.map(|cover| IndexRows {
indexed: cover.indexed_rows,
unindexed: cover.unindexed_rows,
}));
}
let indices = table.dataset.load_indices().await?;
if indices.is_empty() {
return Ok(None);
}
let mut worst = 0u64;
let mut worst = IndexRows {
indexed: 0,
unindexed: 0,
};
let mut names: Vec<String> = indices.iter().map(|index| index.name.clone()).collect();
names.sort();
names.dedup();
@@ -88,15 +108,27 @@ async fn unindexed_rows(table: &OpenTable) -> Result<Option<u64>> {
let raw = table.dataset.index_statistics(&name).await?;
let stats: serde_json::Value = serde_json::from_str(&raw)
.with_context(|| format!("parse index statistics for {name}"))?;
let unindexed = stats
.get("num_unindexed_rows")
.and_then(|value| value.as_u64())
.unwrap_or(0);
worst = worst.max(unindexed);
let count = |key: &str| stats.get(key).and_then(|value| value.as_u64()).unwrap_or(0);
if count("num_unindexed_rows") >= worst.unindexed {
worst = IndexRows {
indexed: count("num_indexed_rows"),
unindexed: count("num_unindexed_rows"),
};
}
}
Ok(Some(worst))
}
/// A vector index also reindexes once a tenth of it is uncovered, so a small
/// index does not wait for the absolute budget.
fn effective_budget(budget: u64, rows: &IndexRows, vector_table: bool) -> u64 {
if vector_table {
budget.min(rows.indexed / 10)
} else {
budget
}
}
#[async_trait]
impl JobHandler for OptimizeIndicesHandler {
fn capability(&self) -> JobTypeCapability {
@@ -138,6 +170,12 @@ impl JobHandler for OptimizeIndicesHandler {
defaults.clone(),
)),
worker_default_values: defaults,
// A retrain is a whole build, so it gets a build's time.
admin_runtime_defaults: Some(AdminRuntimeDefaults {
job_type_max_runtime_seconds: super::vector_index::EXECUTION_TIMEOUT_SECONDS,
execution_timeout_seconds: super::vector_index::EXECUTION_TIMEOUT_SECONDS,
..Default::default()
}),
..Default::default()
}
}
@@ -162,7 +200,7 @@ impl JobHandler for OptimizeIndicesHandler {
let mut proposals = Vec::new();
for encoded in &tables {
let id = parse_id(encoded);
let table = match dataset::open(&client, &id, &self.fallback).await {
let table = match dataset::open(&client, &id, &self.storage).await {
Ok(table) => table,
Err(err) => {
if let Some(counters) = &self.metrics {
@@ -175,8 +213,9 @@ impl JobHandler for OptimizeIndicesHandler {
if let Some(counters) = &self.metrics {
counters.worker.object_seen(JOB_TYPE);
}
let unindexed = match unindexed_rows(&table).await {
Ok(Some(unindexed)) => unindexed,
let vector_table = is_vector_table(&id);
let rows = match index_rows(&table, vector_table).await {
Ok(Some(rows)) => rows,
Ok(None) => continue,
Err(err) => {
if let Some(counters) = &self.metrics {
@@ -186,11 +225,17 @@ impl JobHandler for OptimizeIndicesHandler {
continue;
}
};
let budget = effective_budget(budget, &rows, vector_table);
let unindexed = rows.unindexed;
if unindexed <= budget {
continue;
}
let mut parameters: HashMap<String, ConfigValue> = HashMap::new();
parameters.insert("table_id".to_string(), string_list(&id));
if vector_table {
// The run may turn out to be a retrain.
super::vector_index::declare_build_runtime(&mut parameters);
}
proposals.push(JobProposal {
proposal_id: format!("{JOB_TYPE}:{encoded}"),
dedupe_key: format!("{JOB_TYPE}:{encoded}"),
@@ -237,28 +282,47 @@ impl JobHandler for OptimizeIndicesHandler {
.ok_or_else(|| anyhow!("job {} carried no table_id", job.job_id))?;
let client = self.client();
let mut table = dataset::open(&client, &id, &self.fallback).await?;
let before = unindexed_rows(&table).await?.unwrap_or(0);
let mut table = dataset::open(&client, &id, &self.storage).await?;
let vector_table = is_vector_table(&id);
let before = index_rows(&table, vector_table)
.await?
.map_or(0, |rows| rows.unindexed);
// A vector index whose table has outgrown its partitions is built
// again with partitions for the rows it now holds.
let retrain =
vector_table && vector_index::retrain_due(&table.dataset, VECTOR_COLUMN).await?;
sender.send_progress(JobProgressUpdate {
request_id: request.request_id.clone(),
job_id: job.job_id.clone(),
job_type: JOB_TYPE.to_string(),
progress_percent: 10.0,
stage: format!("indexing {before} uncovered rows"),
stage: if retrain {
"retraining the vector index".to_string()
} else {
format!("indexing {before} uncovered rows")
},
..Default::default()
})?;
// Merging every delta keeps read latency from drifting as the table is
// reindexed again and again; leaving them unmerged is how an index ends
// up fast to write and slow to search.
table
.dataset
.optimize_indices(&OptimizeOptions::default())
.await
.with_context(|| format!("optimize indices of {}", table.location))?;
if vector_table {
vector_index::optimize(&mut table.dataset, VECTOR_COLUMN, retrain)
.await
.with_context(|| format!("optimize the vector index of {}", table.location))?;
} else {
// Merging every delta keeps read latency from drifting as the table
// is reindexed again and again; leaving them unmerged is how an
// index ends up fast to write and slow to search.
table
.dataset
.optimize_indices(&OptimizeOptions::default())
.await
.with_context(|| format!("optimize indices of {}", table.location))?;
}
let after = unindexed_rows(&table).await?.unwrap_or(0);
let after = index_rows(&table, vector_table)
.await?
.map_or(0, |rows| rows.unindexed);
if let Some(counters) = &self.metrics {
counters.rows_indexed.inc_by(before.saturating_sub(after));
}
@@ -268,6 +332,10 @@ impl JobHandler for OptimizeIndicesHandler {
int_value(before as i64),
);
output.insert("unindexed_rows_after".to_string(), int_value(after as i64));
output.insert(
"retrained".to_string(),
seaweed_worker_core::config_form::bool_value(retrain),
);
sender.send_completed(JobCompleted {
request_id: request.request_id.clone(),
@@ -284,3 +352,32 @@ impl JobHandler for OptimizeIndicesHandler {
Ok(())
}
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn a_retrain_gets_a_builds_execution_timeout() {
let descriptor = OptimizeIndicesHandler::new(String::new()).descriptor();
let defaults = descriptor.admin_runtime_defaults.unwrap();
assert_eq!(defaults.execution_timeout_seconds, 3600);
assert_eq!(defaults.job_type_max_runtime_seconds, 3600);
assert!(!defaults.enabled);
}
#[test]
fn vector_tables_reindex_at_a_tenth_of_their_rows() {
let small = IndexRows {
indexed: 60_000,
unindexed: 0,
};
assert_eq!(effective_budget(100_000, &small, true), 6_000);
assert_eq!(effective_budget(100_000, &small, false), 100_000);
let large = IndexRows {
indexed: 5_000_000,
unindexed: 0,
};
assert_eq!(effective_budget(100_000, &large, true), 100_000);
}
}
Loaded 100 of 811 files, more files were not shown because too many files have changed in this diff. Show more