Compare commits

..
Author SHA1 Message Date
949952f2b6 filer: take a hard link off only after the store dropped the name (#12853)
* filer: take a hard link off only after the store dropped the name

Deleting one name of a hard-linked file took the name off the file's link
count (DeleteHardLink: the shared record read, its count less one written
back, in writes of their own) before the store deleted the name. When the
store then refused the delete (a TiKV write conflict, a database timeout),
or the filer stopped in between, the name stayed with the count already one
short. Each time the delete was sent again it took one more off: a client
retrying, or rm run again after the error. At zero the record went, and that
delete, having read a count of one, deleted the chunks. The file's other
names were left with no record and with their chunks deleted. Without a
retry, the next delete of another name of the file did the same.

A name now comes off the count once the store no longer holds it
(FilerStoreWrapper.dropLink). A refused delete leaves the name and the count
as they were, and the delete sent again takes it off once.

- DeleteEntry and DeleteOneEntry delete the name, then drop its link.
- InsertEntry and UpdateEntry over a name of another hard-linked file write
  the new entry, then drop the replaced name's link (handleUpdateToHardLinks
  now returns it instead of dropping it).
- A recursive delete drops its hard-linked children's links after
  DeleteFolderChildren deleted them, not while listing them.
- A link that cannot be dropped after the delete or the write is logged and
  fails neither (a write failed before): the count stays one too high, and
  the record and the chunks outlive the file's last name instead of going
  before it.

Inside a store transaction (a rename) the order changes nothing.

Tests: filer_hardlink_delete_test.go, and through the DeleteEntry handler
filer_grpc_server_hardlink_delete_test.go. With the old order the refused
delete, write and recursive delete each take a link off, and the delete sent
again queues chunk 3,0c03 for deletion while /dir/other.txt reads it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* filer: delete a hard-linked name's chunks only once its file's record is gone

DeleteEntryMetaAndData deleted the chunks of a hard-linked name when the
link count it read before the delete was one or less. Another filer can
link the file under a new name in between: the count misses that name, the
record, which counts it, outlives the delete, and the chunks went while the
new name read them.

Now the count read before must leave no other name and the file's record
must be gone after the delete (lastLinkDropped); a recursive delete already
asks for the record. The count stays in: one name deleted at once through
two filers is taken off twice, and the record goes while another name still
holds the file. When both deletes read the count before either took a link
off, the count still shows that name and the chunks stay. Where the two
disagree the chunks leak rather than go.

Tests: TestDeleteEntryKeepsChunksOfLinkAddedMeanwhile fails with the count
alone, TestDeleteEntryTwiceAtOnceKeepsChunks with the record alone.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* filer: delete a recursive delete's hard-linked names one by one, as a single delete does

A recursive delete listed a folder's hard-linked names, had
DeleteFolderChildren delete them with the other children, and then took
their links off and asked whether each record was gone. Found in review:

- A store that deletes a folder's children in batches (TiKV) can fail after
  some of them. The names it deleted are gone, their links were never taken
  off, and the delete run again no longer lists them, so their counts stay
  high and the file's chunks are never freed.
- Two filers deleting the same folder at once both take the link of each
  hard-linked name off. The second finds the record gone and deletes the
  chunks while a name outside the folder still reads them (as on master).
- The listed entries, chunk lists included, were kept until the folder's
  children were deleted.

A hard-linked child now goes as it is listed, the way a single delete takes
a name (deleteLinkedName): its entry read again for the current count, the
name deleted by the store, then its link taken off (dropLink), and its
chunks queued only when it was the last link (lastLinkDropped: the count
read left no other name and the record is gone). DeleteFolderChildren takes
the other children. Names of one file in two folders of the tree still free
the chunks once, with the last of them.

The link of a hard-linked child now comes off whether or not the delete
removes data, as it already did for a single delete. OnHardLinkIdsFunc and
maybeDeleteHardLinks, which took links off only with the data, are gone.

Tests: TestRecursiveDeleteFailingAfterItsLinkedChildTookItsLinkOff and
TestRecursiveDeletesAtOnceKeepChunksOfANameOutside fail with the previous
commit; the latter also fails when deleteLinkedName asks for the record
alone. TestRecursiveDeleteOfEveryNameDeletesChunksOnce covers the names of
one file in two folders of the tree.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* filer: delete a recursive delete's last-link chunks as the link goes

deleteLinkedName queued the chunks of a file whose last link it took off
with the folder's other chunks, which are deleted only once the whole folder
delete has succeeded. When a later child, a listing or DeleteFolderChildren
failed, the delete returned first and those chunks stayed for good: their
name and record were gone, so no retry could find them. Found in review.

They are now deleted when the link goes, through DeleteChunks with the
child's path, as a single delete deletes them.

Test: TestRecursiveDeleteFailingAfterALastLinkWentDeletesItsChunks fails
with the previous commit.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* filer: free a swept hard-linked child's chunks only when its last link went

The replay sweep freed a hard-linked child's chunks once the shared record
was gone. One name deleted at the same time through another filer takes the
link off twice, so the record goes while another name still reads the file.
Use lastLinkDropped with the count read under the child's lock, as the
single and recursive deletes do.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* filer: test store hands out a copy of an entry, as a real store does

Returning the stored entry let a later lookup overwrite the link count an
earlier lookup handed to its caller.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* filer: look a hard-linked name up again before taking its link off

A delete read the name, and another filer deleted the same name and took
its link off before this delete read the count. The count then already
lacked the name, the second delete took the link off again, the record
went with another name left, and the chunks were deleted. A hard-linked
name is now deleted through DeleteEntry, which looks the name up after
the count was read and skips the link when the name is gone.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* filer: report a failed lookup from a store delete instead of success

DeleteEntry took any lookup error for a name already gone, so the delete
reported success, the name stayed, and the caller sent a delete event.
Only a missing name is a no-op now.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
2026-11-20 21:21:00 -08:00
Chris LuandClaude Opus 5.5 f7bd0ad802 storage: measure the search lock-hold budget in rounds (#12854)
A stall of a loaded host (a goroutine not scheduled for most of a
second) made one search and one foreground write go over the 250 ms
budget, failing the test though every search held the lock for about
20 ms. Measure in five 500 ms rounds and pass once a round stays within
the budget: every search scans the tombstone run, so a slow search still
fails every round.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-20 20:37:19 -08:00
Chris LuandClaude Opus 5.5 3bc6161401 helm: opt-in HTTPS for the volume servers' data port (#12843)
* helm: render volume server HTTPS in security.toml

global.seaweedfs.volumeHttps.enabled sets [https.volume] to the volume
certificate and turns on [https.client] with the client certificate and the
chart CA, so the volume data port is no longer plaintext under
enableSecurity. The option fails without enableSecurity.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* helm: add the volume service names to the volume certificate for HTTPS

Volume servers register as <pod>.<volume service>.<namespace>, and HTTPS
clients verify that name, which none of the certificate's wildcards cover.
With volume HTTPS on, the certificate also lists every volume group's
service.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* helm: probe volume servers over HTTPS when they serve it

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* helm: reject volume HTTPS for the all-in-one deployment

Its volume server registers by pod IP, so HTTPS clients could not verify
the volume certificate.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* helm CI: check the volume server HTTPS rendering and its guards

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* helm: document volume server HTTPS

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* filer: copy chunks for cp.from through the shared HTTP client

The copy built a bare client and plain http:// chunk URLs, so with
[https.client] on it could not read from or write to HTTPS volume servers
("source returned status 400"). Use the global client's transport and
redirect policy, and normalize the source and destination URLs to its
scheme.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* helm: restart every security.toml reader when volume HTTPS is toggled

security.toml is mounted with subPath and read once at startup, so on an
upgrade that turns volume HTTPS on, the volume servers restarted with TLS
while filer, S3 and the other clients kept speaking plain HTTP. A
checksum/volume-https pod annotation, rendered only while the option is
on, rolls them all.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* helm CI: check the volume HTTPS rollout annotation

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* master: forward to the leader with the masters' own HTTP scheme

A follower took the scheme from [https.client], so with HTTPS clients
toward volume servers and plain-HTTP masters every forwarded request
(/dir/assign, /dir/status, the UI) failed with 502 "first record does not
look like a TLS handshake". Masters share one [https.master] setting, so
use HTTPS exactly when this master serves it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* helm: update the volume HTTPS caveats now that masters forward correctly

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* filer: send a read token on the cp.from source chunk GET

With jwt.signing.read set, volume servers refuse unsigned reads, so
cp.from of a chunked file failed with "source returned status 401".
Mint a read token for the chunk on each source attempt, the same way the
filer's own reads do.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* volume (rust): look up volume locations over the master's gRPC port

The lookup used the master's HTTP /dir/lookup with the [https.client]
scheme, so with HTTPS volume servers and plain-HTTP masters every
replicated write, proxied read, remote chunk read and batch delete failed.
Go asks the master's LookupVolume over gRPC; do the same, over one reused
channel.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* volume (rust): bound the whole master dial in lookup_volume

connect_channel's connect timeout covers only the TCP connect, so a master
that accepted the connection and stalled in the TLS handshake held every
replicated write, proxied read and remote chunk read that needed a lookup.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-20 19:20:23 -08:00
github-actions[bot] afdb32d228 docs: regenerate star history chart 2026-11-21 00:51:16 +00:00
Martijn HoekstraandClaude Opus 5.5 1c04179953 s3api: refuse SigV2, SSE-C and SHA-1 checksums under fips140=only (#12845)
With GODEBUG=fips140=only, MD5, SHA-1 and HMAC-SHA1 panic, so a request
that uses SigV2 (header, presigned or POST policy), SSE-C (whose customer
key check is MD5) or the SHA-1 flexible checksum panics in the handler and
the client sees a dropped connection. Refuse them before any hashing:

- SigV2: InvalidRequest "The authorization mechanism you have provided is
  not supported. Please use AWS4-HMAC-SHA256." (ErrSignatureVersionNotSupported)
- SSE-C, including copy-source SSE-C headers: NotImplemented
- x-amz-checksum-sha1 (value header, algorithm headers or query, and
  x-amz-trailer): NotImplemented

fips140Enforced is a package variable so tests can turn the refusals on
without running the test binary under fips140=only. Without enforcement
nothing changes.

Refs #12839

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-20 13:39:12 -08:00
Chris LuandClaude Opus 5.5 0de9a82dd9 filer: make the stub-backed remote storage tests re-runnable (#12852)
filer: drop cached remote clients when a test registers a stub maker

The remote client cache is keyed by the configuration, so rerunning a
stub-backed test (go test -count>1) got the previous run's stub from the
cache and its counters never moved. Nineteen tests failed on a rerun.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-20 13:09:36 -08:00
Chris LuandClaude Opus 5.5 228ccd9eb6 admin: bulk table delete and force namespace delete for S3 Tables (#12851)
* admin: force-delete an S3 Tables namespace with its tables

DELETE /api/s3tables/namespaces?force=true deletes every table in the
namespace, paging through ListTables, before deleting the namespace.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* admin: offer deleting a namespace's tables in the S3 Tables namespace delete dialog

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* admin: select several S3 Tables tables and delete them at once

The tables list gets a checkbox column with select-all and a Delete
Selected button that deletes each selected table after one confirmation.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* admin: clear restored table selections and keep the bulk delete result on screen

Browsers restore checkbox state on reload, which left rows that took the
deleted tables' places selected. The admin alert() is a non-blocking modal,
so reload only after it is closed instead of discarding the failure list.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* admin: re-enable the bulk delete confirm button on page load

Firefox restores a button's disabled state across reloads, which left the
confirm button of the next bulk delete disabled.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* admin: stop a bulk table delete when its dialog is cancelled

Cancel closed the dialog while the remaining deletes kept running. The
loop now stops before the next table and reports how many were deleted.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* admin: say when the S3 Tables list is cut at the list limit

The tables page shows at most the admin list limit, so select-all covers
only the displayed rows; mark a truncated list and name select-all's scope.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-20 13:08:16 -08:00
Chris LuandClaude Opus 5.5 865a7bd988 filer: commit an HTTP append on a non-owner through the entry's owner (#12849)
* filer: commit an HTTP append on a non-owner through the entry's owner

An HTTP append or offset write re-read the entry under its path lock and
cluster lock and then committed with an unconditional upsert. The path
lock is per process and the cluster lock is a lease, so with several
filers on one store a write landing between the re-read and the upsert
was overwritten although both requests were acknowledged.

A filer that is not the entry's write owner now sends the commit to the
owner as a CreateEntry conditioned on the entry it read (IF_ENTRY_EQUAL,
absent when there was none), evaluated under the owner's path lock. The
owner keeps committing locally under that same lock. A failed condition
refuses the append as a changed entry; any other routed error keeps the
uploaded chunks the stored entry references.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* filer: test an HTTP append racing another filer's append

Two filers on one store share a two-member ring. An append on the
non-owner gets another filer's append injected at the store after each
of its reads of the entry in turn; it must either keep both appends or
be refused. Without the routed conditional commit, injections after the
re-read are overwritten.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* filer: keep the chunks of a forwarded append whose outcome is unknown

The owner may have committed before its reply was lost, and with
separate per-filer stores the local lookup may not see that commit yet,
so deleting the chunks it does not find can strip a committed append.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* filer: keep fsync appends on the local commit

A per-request fsync is a local context mark that a forwarded CreateEntry
cannot carry, so the owner would store the append without flushing.
Storage-rule fsync is still applied on the owner.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* filer: normalize the stored entry too when matching IF_ENTRY_EQUAL

A store that keeps the entry as given (the JSON-backed ones) returns
FileSize zero beside chunks, while the expected entry was normalized to
the chunk extent, so an unchanged entry failed the match.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* filer: carry a create's fsync request in CreateEntryRequest

The per-request fsync is a context mark on the receiving filer, so a
create forwarded to another filer lost it. CreateEntry now applies the
request's fsync flag to its store write.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* filer: route fsync appends to the owner with the fsync flag

A fsync append on a non-owner committed locally without a condition, so
it could still overwrite a write the owner committed after its re-read.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* filer: test an offset write on a store that keeps FileSize as given

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* filer: start an append at the entry's full size

A store that keeps the entry as given can return FileSize zero beside
chunks, so the append chunks were placed at offset zero over the
existing bytes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* filer: test an append on a store that keeps FileSize as given

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-20 13:07:18 -08:00
Chris LuandClaude Opus 5.5 137e4aeede s3api: keep the write condition on the lock fallback create (#12846)
* s3api: keep the write condition on the lock fallback create

Without a ring the PUT checks its conditional headers at the gateway and
then sends the create unconditionally under the object write lock. That
lock is a short lease, so a paused or slow holder lets a second
conditional writer in, and both writes succeed. Send the reduced
condition with the create so the filer re-checks it under its path lock.
Suspended-versioning writes keep the gateway check, since their condition
names the latest version rather than the create's path.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3api: test two If-None-Match PUTs racing on the lock fallback

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3api: keep the chunks of a fallback create whose outcome is unknown

A conditional create is forwarded by the filer to its ring owner, which
a gateway without a ring may not know, so its landed-check can miss a
commit the owner made before the response was lost. Reclaim the chunks
of such a create only when the filer refused it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3api: test exact fallback race results and a lost conditional create

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3api: reclaim the chunks of a fallback create the filer refused

A precondition refusal proves the create never committed, but the
landed-check finds the winning writer's entry and reports the key as
present, so the refused writer's chunks were left orphaned. The
ambiguous-route recovery keeps its own rules.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3api: test that the refused fallback writer's chunk is reclaimed

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3api: keep the chunks of a recovery create whose outcome is unknown

The owner can forward the conditional recovery create of an ambiguous
routed PUT to a filer the gateway cannot look up, so the cleanup gate
now follows the condition each create actually sent.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3api: test that the fallback race deletes only the refused writer's chunk

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-20 13:06:24 -08:00
Chris Lu 0c705ec9f2 pb: client interceptors on every dialed connection (#12834)
AddClientUnaryInterceptor registers an interceptor that runs around every
unary call on a connection opened by GrpcDial. GrpcDial installs one
fixed interceptor that consults the registered list at call time, so one
added later also applies to connections already open and cached. With
none registered it only calls through.

Claude-Session: https://claude.ai/code/session_011Upvag1tiz2pnSB8mS2qtK
2026-11-20 13:05:23 -08:00
037c27582f redis2: let Redis hash the Lua scripts (no SHA-1 at package init) (#12848)
* redis2: let Redis hash the Lua scripts (no SHA-1 at package init)

redis.NewScript computes the script's SHA-1 in Go when the package-level
vars are initialized. redis2 is linked into every weed binary, so under
GODEBUG=fips140=only every subcommand, including `weed version`, panics
before main. NewScriptServerSHA leaves the digest to Redis (SCRIPT LOAD
on first use) and keeps Run's EVALSHA-then-EVAL behaviour.

weed/fips140_only_test.go runs the weed main package's test binary with
fips140=only, so any package init that uses an unapproved algorithm
fails `go test ./weed/...`.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* redis2: run the Lua scripts by a pinned digest (no SCRIPT LOAD, no lock)

NewScriptServerSHA gets the digest with SCRIPT LOAD inside ensureHash, which
holds the Script's write lock across the round trip and does not remember a
failed load. Where an ACL denies SCRIPT but allows EVAL, every call retried
SCRIPT LOAD under that lock before falling back to EVAL.

The two scripts are now constants next to their SHA-1 digests, and runScript
sends EVALSHA and falls back to EVAL on NOSCRIPT, the same commands
Script.Run sent on master. No SHA-1 is computed in Go, at init or at run
time. TestScriptDigests checks each pinned digest against its source.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* redis2: hash the script digests in the test without FIPS enforcement

TestScriptDigests panicked under GODEBUG=fips140=only, the mode this
change is for, because sha1.Sum is not approved there.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
2026-11-20 13:03:47 -08:00
1b209f7717 util, s3api: AES-GCM via NewGCMWithRandomNonce for fips140=only (#12847)
* util: AES-GCM via NewGCMWithRandomNonce in Encrypt/Decrypt

cipher.NewGCM returns an error under GODEBUG=fips140=only ("use of GCM
with arbitrary IVs is not allowed"), so filer -encryptVolumeData cannot
encrypt or decrypt chunks in that mode. NewGCMWithRandomNonce produces the
same nonce(12) || ciphertext || tag layout, so chunks written by earlier
versions still decrypt and older versions can read new chunks; the tests
check both directions against a copy of the previous code.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3api: SSE-S3 DEK and KEK wrapping via NewGCMWithRandomNonce

cipher.NewGCM returns an error under GODEBUG=fips140=only, so SSE-S3
PutObject fails in that mode. NewGCMWithRandomNonce yields the same nonce
and ciphertext+tag bytes: the wrapped KEK keeps its magic || salt || nonce
|| ciphertext+tag layout, and the DEK's nonce and ciphertext are still stored
as separate metadata fields. sealGCMSplit and openGCMSplit split and join
them, with the nonce length as gcmNonceSize, because the random-nonce AEAD's
NonceSize() reports 0.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* util, s3api: keep AES-GCM working for AES blocks NewGCMWithRandomNonce refuses

cipher.NewGCMWithRandomNonce accepts only the standard library's own
*aes.Block. In a GOEXPERIMENT=boringcrypto build aes.NewCipher returns
BoringCrypto's block instead, so Encrypt, Decrypt and SSE-S3 key wrapping
would fail there. util's newGCMWithRandomNonce falls back to cipher.NewGCM
with a nonce from crypto/rand for such blocks, with the same
nonce || ciphertext || tag output. SSE-S3's KEK wrap and DEK wrapping use no
additional data, so they now go through util.Encrypt and util.Decrypt and
build no AES-GCM of their own; the stored bytes are unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* util: keep the GCM fallback test working in boringcrypto builds

cipher.NewGCMWithRandomNonce refuses aes.NewCipher's block in a
GOEXPERIMENT=boringcrypto build, so the test failed there before it
reached the fallback. Compare with cipher.NewGCM and the nonce in front
when the native constructor is unavailable.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3api: build the SSE-S3 sealed buffers with slices.Concat

Summing the input lengths for make() trips CodeQL's
go/allocation-size-overflow on the DEK path, whose inputs come from
stored metadata.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3api: draw SSE-S3 key IDs from crypto/rand

The math/rand key ID is serialized next to the wrapped DEK, so CodeQL's
go/insecure-randomness follows it from the metadata JSON into the
AES-GCM open of the DEK.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
2026-11-20 13:03:22 -08:00
Chris LuandClaude Opus 5.5 f69cb98042 admin/plugin: extension points for scheduling policy (#12833)
* admin/plugin: extension points for scheduling policy

The lane scheduler decides four things in line: how a lane's job types
are run, whether a proposal becomes a queued job, whether a queued job
starts on the slot reserved for it, and whether a job holds the cluster
admin lock for its whole run. Each now goes through a schedulerExtension
whose default answers exactly as before, so policy can be added without
editing the dispatch loop. The extension can also serve its own gRPC
service beside the worker stream.

Claude-Session: https://claude.ai/code/session_011Upvag1tiz2pnSB8mS2qtK

* admin/plugin: state when runLane may return and when done runs

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-20 13:02:14 -08:00
Chris LuandClaude Opus 5.5 22df63ad22 security: accept a previous JWT signing key during rotation (#12844)
* security: verify JWTs against an optional previous signing key

DecodeJwt, the admin token checks and the volume expiry classifier take
previous keys to try after the current one, and Guard keeps a previous
write and read key next to the current ones. Signing still uses only the
current key. Callers pass no previous key yet.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* volume, filer, s3, admin: accept tokens signed with jwt.*.previous_key

jwt.signing.previous_key, jwt.signing.read.previous_key,
jwt.filer_signing.previous_key and jwt.filer_signing.read.previous_key
(WEED_JWT_*_PREVIOUS_KEY from the environment) name a key the verifiers
still accept next to the current one, and SIGHUP reloads them. Keys can
then rotate without an error window: add the new key as previous
everywhere, swap current and previous, then remove previous.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* scaffold: document jwt previous_key for key rotation

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* volume(rs): accept tokens signed with jwt.signing previous_key

Mirrors the Go volume server: [jwt.signing] and [jwt.signing.read] take
previous_key (WEED_JWT_SIGNING_PREVIOUS_KEY,
WEED_JWT_SIGNING_READ_PREVIOUS_KEY), the guard verifies against it after
the current key, and SIGHUP reloads it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* volume, filer, s3: drop a removed previous_key on SIGHUP

LoadConfiguration merges, so a previous_key line deleted from
security.toml kept verifying until a restart. Read the previous keys
without the merge on reload, as the Rust volume server already does.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* volume, filer, s3: keep previous keys when a reload cannot read the file

A SIGHUP that found security.toml unreadable or malformed cleared the
previous keys while the merged current keys stayed. Fall back to the
merged configuration then, so the keys in effect stay in effect.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* filer: IAM and metadata subscription admin checks follow the filer guard

The IAM gRPC service checked the jwt.filer_signing keys read at startup,
and the metadata subscription read them from the merged configuration,
so adding or removing previous_key with SIGHUP did not reach them. Both
now read the filer guard that Reload updates.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* security: reload jwt keys in one place, keeping the guard's previous keys

When security.toml could not be read, the reloads fell back to the merged
configuration, which still holds a previous_key an earlier reload removed.
Guard.ReloadSigningKeys keeps the previous keys the guard already has
instead, and the volume, filer and S3 reloads share it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* security: take reloaded current keys from the same read as previous keys

The current keys came from the merged configuration and the previous keys
from a second read of the file, so a file swapped between the two reads
could publish the same key as both. Take every key the fresh read sets
from it; a current key the file no longer sets keeps its merged value.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* admin: worker streams read the filer admin keys as security.toml sets them

Both worker stream handlers read jwt.filer_signing keys from the merged
configuration. Under weed mini a reload merges the file, which keeps a
previous_key the operator removed, so new streams still accepted it; a
standalone admin never picked up a rotation at all. Read the keys the file
sets now, falling back to the merged values.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* scaffold: wait for old-key tokens to expire before clearing previous_key

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* volume(rs): keep the keys in effect when SIGHUP cannot read security.toml

parse_security_config falls back to defaults when the file is missing or
unreadable, so a SIGHUP then cleared every key and turned JWT checks off.
Skip the key and whitelist update in that case, as the Go reload keeps
its merged configuration and the guard's previous keys.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* security: keep every key in effect when a reload cannot read security.toml

A failed read took the current keys from the merged configuration while
keeping the active previous keys, so after a swapped pair the guard could
publish the old key twice and reject tokens signed with the new one. The
reload now leaves the keys untouched, as the Rust volume server does.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* security: filer admin keys fall back to the last pair read, not merged values

When security.toml stopped being readable, FilerAdminKeys returned the
merged configuration, which still holds a previous_key the file had
already dropped, so tokens signed with the retired key verified again.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* filer: STS store gates check the admin token against one key snapshot

The OIDC provider and role RPCs read the reloadable admin key twice, in
checkAdminAuth and in the store gate. A reload that set the key between
the two reads let a call without a token pass both. The gate now takes
one snapshot for the empty-key refusal and the token check.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* security: read and cache filer admin keys under one lock

Concurrent callers could store their reads out of order, so an older
pair that still had previous_key could overwrite a newer one and come
back when a later read of security.toml failed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* security: unlock the filer admin key mutex on every exit of its test

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-20 13:01:27 -08:00
Chris LuandClaude Opus 5.5 4fefe7863c scaffold: list every component that needs each JWT signing key (#12838)
* scaffold: list every component that needs each JWT signing key

jwt.signing.read.key is not a master and volume only key: the filer, the
S3 gateway, mount, webdav, sftp, filer.sync/backup, workers, admin and
weed shell all sign their own volume read tokens with it. Giving the S3
gateway only the filer keys made every GET fail with 401 wrong jwt. Drop
the stale note that the filer does not support read JWTs, and name the
filer and weed shell as signers of the volume write key.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* scaffold: name admin and workers as filer signing key users

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-20 12:59:59 -08:00
Chris LuandClaude Opus 5.5 234886d801 admin/plugin: apply the completion of a job whose progress carried no state (#12828)
* admin/plugin: keep a job running when its progress has no state

A progress update without a state stored job_state_unspecified on the
tracked job. That state is not active, so attemptEnded took the attempt
for closed and dropped the worker's JobCompleted: the job stayed at its
first progress stage, with no run record and no completion activity.
seaweed-worker sends its progress without a state.

Treat an unspecified progress state as running.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* seaweed-worker: report progress as running when the handler sets no state

The Lance handlers send their progress without a state, so admin saw
JOB_STATE_UNSPECIFIED where the Go handlers send JOB_STATE_RUNNING, and
dropped the job's completion. Default the state in the stream sender so
every handler reports like the Go ones.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-20 12:59:27 -08:00
Chris LuandClaude Opus 5.5 425216714c helm: stronger JWT signing keys, keep security config in a Secret (#12836)
* helm: generate 64-character JWT signing keys

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* helm: keep security.toml in a Secret

The JWT signing keys were readable by anyone allowed to read ConfigMaps
in the namespace. Existing keys are carried over from the ConfigMap on
upgrade.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* helm: document where the security config is kept

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* helm: test upgrading a release of the ConfigMap chart

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* helm: refuse a partitioned upgrade that removes the security ConfigMap

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* helm: check for the security ConfigMap even when the Secret exists

A failed migration leaves the new Secret next to the old ConfigMap, so a
partitioned retry must still be refused before Helm removes the ConfigMap.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* helm: test a partitioned retry after a failed migration

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* helm: guard partitioned upgrades only for this release's ConfigMap

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* helm: test a partitioned upgrade next to another release's legacy ConfigMap

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-20 12:58:30 -08:00
Chris LuandClaude Opus 5.5 d68cde4cfc s3api: sign the virtual-hosted bucket in signature V2 (#12835)
* s3api: sign the virtual-hosted bucket in signature V2

Signature V2 signed only the request path, so a path-style signed or
presigned request replayed with a virtual-host Host header reached
another bucket. Prefix the bucket taken from the Host to the canonical
resource, as AWS does, and refuse a request routed to a different bucket.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3api: test signature V2 against virtual-host retargeting

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3api: match port-qualified virtual-host domains like the router

With -s3.domainName=s3.test:8333 the router matches the full host:port,
but bucketFromVirtualHost compared the port-stripped host and never found
the bucket, so the virtual-host checks did not apply.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-20 12:55:03 -08:00
Chris LuandClaude Opus 5.5 7d52383ccf admin: show tooltips on the S3 Tables namespaces and tables buttons (#12837)
* admin: title the S3 Tables namespaces and tables buttons

The icon-only links that open a bucket's namespaces and a namespace's
tables had no title, so hovering them showed no tooltip while the other
row actions did.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* admin: title the topic details button

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* admin: switch the topic details button title with its state

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-20 12:53:20 -08:00
3aadc0fc46 s3api: bound route-by-key lookups at an unreachable owner filer (#12827)
* s3api: bound route-by-key lookups at an unreachable owner filer

A route-by-key read (getObjectEntryRoutedByKey and the scoped .versions
listing) asks the key's owner filer first, on context.Background(). When
that owner sits behind a network partition that drops packets instead of
refusing them, the lookup waits out gRPC's 20 s connect timeout before the
failover walk asks the next filer. The owner is then skipped for only 2 s,
so the next read of any key it owns dials it again and waits another 20 s.
With the ring spread over two data centers, about half the keys stall
every ~22 s for as long as the partition lasts (#12826).

- The owner's attempt in the walk now gets a deadline,
  -s3.routedLookupTimeout (default 1s), when the owner is not the
  gateway's current filer. Past it the walk moves on to the next filer, as
  for any transport error, and flags the owner unreachable. The current
  filer and every later attempt stay unbounded, so a slow but reachable
  local filer still answers; 0 restores the unbounded lookup. The prior-
  owner probe during a ring rebalance gets the same bound.
- The skip time of a failed owner is now -s3.unreachableOwnerTTL
  (default 2s, unchanged). Deployments that can see dropped-packet
  partitions can raise it so each owner is retried less often.
- The S3 gateway's volume location lookups take -s3.volumeLookupTimeout
  per filer (default 5s, the FilerClient default, unchanged), so a vidMap
  miss need not wait 5 s per unreachable filer.

The flags exist on weed s3 (without the s3. prefix), weed filer, weed
server and weed mini.

Tests (s3api_object_routed_read_deadline_test.go): an owner that never
answers costs a read the deadline and is skipped by the next read (hangs
without the change); with the deadline 0 a slow owner is still waited
for; an owner that is the current filer is not timed out; a slow filer
later in the walk is waited for; the scoped listing fails over at the
deadline; the skip TTL honours the option and the default.

* s3api: bound a routed listing only until the owner's first message

The routed-lookup deadline covered the whole .versions stream at a remote
owner. A reachable owner whose listing keeps streaming past it (a store
scan trailing its first rows) was cut off mid-stream, and the failover
walk refuses to replay a stream that already delivered entries, while
listRetry does not retry DeadlineExceeded, so the listing failed outright
where it used to succeed. Lift the deadline once the stream delivers its
first message; an owner that never answers still fails over at it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-20 12:52:30 -08:00
小马一只andClaude Opus 5.5 aa95f631f2 http: keep the body pieces that arrive after a read is cancelled (#12822)
RetriedFetchChunkData's callback for ReadUrlAsStream skipped every piece
once ctx was cancelled. ReadUrlAsStream checks ctx only before each body
read, so when ctx is cancelled while the last read is under way, that
read returns the last piece with io.EOF, the callback drops it, and the
fetch returns fewer bytes than asked with a nil error. Nothing above it
checks the length: weed mount answers the FUSE read with the short
buffer, and the kernel takes a short buffered read as the end of the
file, so the reader sees EOF in the middle of it.

The callback now keeps every piece. A read cancelled before its last
piece still ends with ctx.Err() from ReadUrlAsStream's own check, so the
download still stops; one cancelled during the last piece returns the
whole range. An encrypted chunk reaches the callback in one piece, so
there the old check lost the whole chunk: 0 bytes and no error.

TestRetriedFetchChunkDataCancelledWhileTheLastPieceArrives cancels ctx
inside the body read that returns the last piece, for a range read and
an encrypted chunk; both fail on master with fewer bytes and no error.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-20 12:50:44 -08:00
Yip LeoandClaude Opus 5.5 5adfc23e85 s3api: serve a 0-byte file on HEAD when its child probe fails (#12824)
HEAD of a 0-byte file lists one child to tell an implicit directory from
an empty file, and hasChildren answers true when that listing fails. The
listing callers want that, since keeping an entry is their safe side, but
HEAD then answers 404 NoSuchKey for a file it has just found. A copy tool
that HEADs the key it just wrote reports the copy as failed; rclone logs
"Failed to copy: object not found" for every empty __init__.py or lock
file whose probe failed, although the object is in place.

The probe fails like this whenever the request context is already
canceled while the client still waits. net/http cancels a connection's
context when its background read fails, and the read deadline the
activity-timeout listener sets can expire during a long request that
neither reads nor writes, such as a large server-side copy. The
connection stays open and keeps serving keep-alive requests, each with a
context canceled from the start, so every 0-byte HEAD on it answered 404.

HEAD now asks zeroByteFileHasChildren, which answers false when the probe
fails, and the file is served. hasChildren keeps answering true for the
listings. A probe on an already-ended context returns at once instead of
failing every filer of the failover walk and counting each as a failure.

TestZeroByteFileHasChildrenServesTheFileWhenTheProbeFails covers a real
child, no child, a canceled context and a filer error.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-20 12:49:44 -08:00
f28d101a82 filer, mount: fail writes to a read-only path at once, as EDQUOT (#12823)
* filer: report a read-only create in the response, not as a gRPC error

CreateEntry returned a storage rule's read-only verdict (a bucket over
quota) as a status-less error, which reaches clients as codes.Unknown.
Clients retry Unknown like a filer outage. A mount retried the create of
every new file four times (1/2/4 s) before returning EIO, so each close()
on an over-quota bucket hung 7 s, git on its index.lock included.

Report it the way AssignVolume already does: a successful RPC whose
response carries the text and FilerError_READ_ONLY. CreateEntryWithResponse
and the mount's stream path already map that code to filer_pb.ErrReadOnly.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* mount: fail a write to a read-only path at once, as EDQUOT

On a read-only path (a bucket over quota), close() of a new file retried
its create for 7 s and then returned EIO, and a failed chunk upload
returned EIO as well. The verdict holds until the bucket is back under
quota, so a retry gains nothing, and EIO names no cause.

- isRetryableFilerError does not retry filer_pb.ErrReadOnly.
- grpcErrorToFuseStatus, writeErrorToFuseStatus, and mkdir, symlink and
  link return EDQUOT for it. ENOSPC still means the cluster is out of space.
- ShouldReassignUpload never reassigns on it, even when the path in the
  message contains "transport".
- CreateEntryWithResponse and the stream path recognize the text an older
  filer sends ("read only: <prefix> ...", as codes.Unknown or as a bare
  message). The mount stops retrying even before its filer is upgraded.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* admin, iam, shell: check a create's response, not only the RPC error

A filer reports a refused create in CreateEntryResponse with a nil RPC
error: an existing directory, a parent that is a file, a store error,
and now a read-only path. Ten callers sent CreateEntry directly and
looked only at the RPC error, so they took every refusal for success. An
admin upload into a bucket that went read-only mid-upload reported
success and skipped its chunk cleanup, and a session revocation that the
filer refused left the session usable.

They now go through filer_pb.CreateEntry, which returns the response's
error as the call's error.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* pb, mount, operation: read a filer verdict as an answer, not a failure

A filer verdict, such as a read-only path, names the path in its
message. Classifiers that fall back to matching that text for words like
"transport" still read a refusal of /buckets/b/transport/f as a broken
network:

- shouldInvalidateConnection closed the shared filer channel, failing
  every other RPC in flight on it.
- The mount's withFilerClient retried the call for about 13 s.

filer_pb.IsFilerVerdict reports whether an error carries a sentinel that
a filer response code maps to. shouldInvalidateConnection keeps the
channel for such an error, and also for an older filer's read-only
refusal that arrives as codes.Unknown. withFilerClient no longer retries
it, and an upload with RetryForever no longer loops on a read-only
assignment.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* mount: keep the first filer's verdict when another filer is down

withFilerClient walks the filer addresses and keeps the last error. When
the first filer refuses a write to a read-only path and the next one is
unreachable, the walk returned the Unavailable from the second, so the
call was retried for 13 s and then failed without the verdict.

Stop the walk at a verdict and keep that filer as the one to try first:
it answered, and every filer gives the same verdict.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* java client: fail a metadata write the filer refuses in its response

The filer now reports a create on a read-only path in CreateEntryResponse
instead of as a gRPC error. writeMeta ignored the response, so closing a
zero-byte file under a read-only rule reported success without creating it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
2026-11-20 12:49:05 -08:00
Chris LuandClaude Opus 5.5 89457b1387 vector: indexed metadata keys, worker job, table state and shell (#12820)
* vector: derive the indexed metadata key columns

An indexed metadata key is a hidden `_mk_<n>` column of List<Utf8>: the
strings an S3 `$eq` on a string matches, that is the value when it is a
string or the string elements of an array, and nothing for any other type
or a missing key. The field metadata names the key and its state
(`building` or `ready`).

The module derives the column from metadata given as JSON (the write
path) or as stored (jsonb or JSON text, for a backfill), and reads the
indexed keys from a schema. It lives in the worker crate so the engine and
the worker share it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* vector: fill the indexed metadata key columns on PutVectors

PutVectors builds its batch against the table's current schema. Once a
table has indexed metadata key columns, the three-column batch no longer
matches that schema and the write fails. Fill each `_mk_*` column the
schema has from the row's metadata, whatever the key's state; a column
the engine does not know still fails the request.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* vector: answer string $eq and $in on a ready indexed key from its column

The gateway renders a string `$eq` or `$in` on a key as an OR of
`json_extract(metadata, P) = '"v"'` (or IN) and
`json_array_contains(metadata, P, '"v"')` over the same values. On the
parsed filter, a top-level conjunct of that shape on a key whose column is
`ready` becomes `array_has` / `array_has_any` on the column, which its
LABEL_LIST index answers. The two halves must name the same key and the
same strings, the scalar half in the text json_extract returns for them;
anything else, including `$ne`, `$nin`, `$exists`, numbers and booleans,
stays on the JSON path, and the other conjuncts still apply.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* vector: score few indexed matches exactly, per scope

With a ready indexed key in the filter, `ann` first counts the rows of its
scope that the key's conjuncts match, through the scalar index. When they
are at most the limit it scores them exactly over that scope, the way
the short-result fallback does; above it, the vector index searches with
the indexed prefilter, and the fallback still applies. The scope is the
one `exact_in_scope` scores, so per-scope exact answers merge into the
global exact top-k.

The limit is `Query::exact_match_limit` (None takes 256; 0 never counts),
set by the engine's --exact-match-limit (WEED_VECTOR_EXACT_MATCH_LIMIT).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* vector: count filtered queries by metadata key

For each index, the engine counts its QueryVectors calls by the metadata
keys of the parsed filter, with their total latency and the operand types
each key was compared with, over the last hour in 5-minute buckets. At most
512 indexes and 32 keys per index are kept (an idle key makes room;
otherwise the index is marked truncated), and the stats are not exported
as metrics labels.

The new FilterStats RPC returns an index's stats; the gateway's discovery
client forwards it like the other calls.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* vector: report the metadata columns capability in gRPC health

An engine whose writes fill the indexed metadata key columns reports the
gRPC health service `weed.vector.capability.metadata_columns` as serving,
so an operator action can refuse to index a key while an older engine
could still write the table.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* vector: expire filter stats' operand kinds and truncation with the window

The operand kinds a key was compared with, and the mark that a key went
uncounted, outlived the queries that set them. Keep the kinds per
5-minute bucket and the truncation as the bucket it last happened in, so
both leave the hour window with the counts.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* vector: fill indexed key columns a write left empty racing their add

A PutVectors that opened the table before an indexed key's column was
added, and commits after the add, is rebased by Lance onto the wider
schema: it commits, and the rows it wrote hold NULL in the new column, so
the column's index misses them (reproduced in the test: 100 of 150 rows).

After its commit, PutVectors now looks for indexed-key columns the batch
did not carry, reads those of its keys whose column is still NULL, derives
the values from their stored metadata and patches just those columns with
a partial-schema merge that updates only rows still NULL.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* vector: keep filter stats' truncation mark at the later bucket

record() reads the clock before it takes the lock, so a call that waited
on the lock across a bucket boundary could move the truncation mark (and
the index's last-used bucket) back to its earlier bucket, and the mark
then left the window one bucket early.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* vector: derive a raced write's indexed key columns from current metadata

A PutVectors that opened the table before an indexed key's column was
added is retried by Lance against the wider schema: rows it inserts hold
NULL there, and rows it updates keep the value derived from their old
metadata, which the NULL-only heal never looked at. The heal now derives
the columns for all of the write's keys.

Its merge also ran with Lance's retry, which joins the values derived
before the conflict onto the rows a later write of the same keys left.
It now fails on a conflict and is redone from the latest version.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* vector: check the raced write's keys keep the later write's rows

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* vector worker: the indexed metadata key lifecycle on the Lance table

A key's whole state lives in the table: a request is schema metadata
(`weed:metadata_key_request:<key>`), the key's `_mk_<n>` column carries
`building`, `ready` or `off`, and its LABEL_LIST index is named
`_mk_<n>_idx`.

- request / request_removal: the cheap metadata-only steps an operator
  action takes. Removal of a column sets it off at once.
- add_column computes the column for every row with add_columns.
- finish fills rows still NULL (a write that raced the add and died
  before filling them), builds the first segments, fills again, marks the
  key ready and clears the request.
- index_fragments keeps the index as time-ordered segments: the first
  build splits the table into segments of up to `segment_rows` rows in
  fragment order (uncommitted builds committed in one CreateIndex);
  later, once unindexed rows reach a threshold, they fold into the head
  (num_indices_to_merge = 1) or start a new head once the head is full
  (num_indices_to_merge = 0).
- maintain fills NULL rows, then folds; remove_column sets off, drops the
  index, then the column.

Filling uses a partial-schema merge that only updates rows still NULL; it
moves the rows it updates to a new fragment, so it serves the few rows a
racing write leaves, not the backfill, which add_columns does in place.

The test runs the whole lifecycle, including a write that opened the
table before the add and committed after it without filling its rows, and
a compaction: Lance keeps each segment's fragments apart and remaps every
segment, and indexed answers match the JSON filter throughout.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* vector: list and request indexed metadata keys through the engine

MetadataKeys reports an index's keys as its Lance table records them:
requested (schema metadata), building, ready or off (the column's field),
with the rows, segments and bytes of each key's scalar index.
UpdateMetadataKey records a request to index a key, or to stop: a request
is withdrawn, a column is set off at once. Both are refused with
FailedPrecondition for a key indexed or requested already, past eight
keys, or not indexed. The gateway's discovery and describe clients
forward both, so the shell and admin reach them like the other calls.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* vector worker: lance_metadata_index job adds, maintains and removes keys

Detection reads each S3 Vectors table's keys: a requested or `building`
key is to add, an `off` one to remove, and a `ready` one whose unindexed
rows reached `min_unindexed_rows` (default 50,000) to maintain. A job
names the table, key and action, as the admin builds it when it
dispatches one for an operator, and does only what the table still asks
for when it runs. `segment_rows` (default 1,000,000) caps a segment.
The job runs in the Lance scheduler lane.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* s3vectors: list and request indexed metadata keys for trusted callers

Server.MetadataKeys and Server.UpdateMetadataKey reach the engine's new
RPCs for the shell and the admin server. Adding a key refuses a
non-filterable key of the index, and refuses unless every vector engine
the masters list reports the gRPC health service
weed.vector.capability.metadata_columns as serving, so no older engine can
write the table without filling the key's column.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* shell: s3vectors.index.metadata lists, adds and removes indexed keys

s3vectors.index.metadata -bucket b -index i [-add key | -remove key]
prints each indexed metadata key with its state, the share of rows its
scalar index covers, its segments and size. The shell's in-process S3
Vectors server setup moves into shellVectorsServer for both commands.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* vector worker: derive raced rows again, not only the NULL ones

A write that raced a key's column add leaves NULL in the rows it
inserted and, in the rows it updated, the value derived from their old
metadata. Filling only NULL rows missed the second kind.

rederive compares each row's column with the value its stored metadata
gives and rewrites the rows that differ, starting over from the latest
version when a write commits meanwhile. finish runs it over the table
before the first build and over the fragments written during the build;
maintain runs it over the fragments the index does not cover yet, before
folding them. The lifecycle test's raced write now moves 500 rows to
another author as well as inserting 500, and all are right once the key
is ready.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* vector: count indexed matches segment by segment, newest first

The planner counted a filter's indexed matches with a Lance scan of the
query's scope. A fragment-scoped scan makes Lance read the column
instead of answering from the index, and it counted every match.

The count now opens each segment of the key's index that overlaps the
scope, newest first, searches up to eight of them at once in order,
keeps the rows of the scope's fragments, and stops as soon as the total
passes the limit. Rows in fragments no segment covers yet are counted by
a read of the column. An AND is bounded by its most selective indexed
conjunct. Rows a segment still lists after a delete count too: the total
is an upper bound, which can only send a query to the vector index
instead of an exact scan.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* vector worker: test jobs on a table dropped under them (fails)

DeleteIndex purges the table directory, and a colocated index created
again under the same name reuses it. The test runs a vector index
build and the metadata key add, maintenance and remove jobs against a
table dropped right after the job opened it, and at the job's first
write. It expects the job to fail, no manifest in the dropped path (no
file at all when dropped before the first write), and a table created
there again to start empty.

It fails today: the remove job's metadata-only commits succeed and write
two manifests into the dropped path, after which creating the table
again fails with "Dataset already exists". The next commit fixes it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* vector worker: a job does not write into a table dropped under it

Jobs open tables through dataset::open, which now wraps the store in a
DropGuard: once the manifest of the version the job opened is gone,
every write fails ("the table was dropped while this job wrote it"),
so the job fails instead of committing a manifest of the old table into
the dropped path. A drop can land between the check and a write, so a
manifest write is checked again afterwards and removed if the table is
gone. Data or index files written in that window stay in the path
unreferenced; no manifest does.

With the guard, the vector index build and the metadata key add,
maintenance and remove jobs fail on a table dropped after their open or
at their first write, leave no manifest (nor any file when dropped
before they wrote), and a table created there again starts empty.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* vector: test an index dropped and created again on one engine (fails)

DeleteIndex purges the table's directory through the catalog and an
index created again under the same name reuses it. On the same running
engine, after a ready indexed key, a query and its filter stats, the
test drops and re-creates the index, writes new rows and queries again:
no old row, indexed key or filter stat may come back.

Rows and keys come back clean; the filter stats of the old table are
still served for the new one. The next commit fixes it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* vector: filter stats follow the table behind an index name

An index dropped and created again under its name is another table, but
its filter stats were kept by name, so the new index showed the old
one's filtered queries. The engine now forgets an index's stats when it
creates or drops the index, and, since DeleteIndex purges through the
catalog without telling the engines, a query of an index whose table id
differs from the one its stats were counted on starts them over.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* s3vectors: refuse indexed key calls when no engine is configured

A Server built without a vector engine client (as the admin's tests
build it) panicked on MetadataKeys instead of answering with an error.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* vector worker: a key removed during its build stays removed

Removing a key whose build had not finished set its column off but left
its add request, which reappeared once the column was dropped and asked
for the key again. A build that finished after the removal also set the
key back to ready.

Removal now withdraws the request too, both when the operator asks and
when the job drops the column, and finish marks the key ready only if
the latest version still has it building.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* vector worker: maintain a key at once when a write left NULL rows

A write that raced a key's add and stopped before fixing its rows leaves
NULL in the key's column, so queries through the key miss those rows.
They land in fragments the index does not cover yet, which maintenance
derives again, but only once the unindexed rows reached the threshold.
Detection now also proposes maintenance when those fragments hold a NULL
row.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* vector worker: test a purge through the drop guard (fails)

A purge removes the versions before its rebuild, including the one it
opened, which the drop guard watches; recording the purge's completion
then fails as if the table had been dropped. The next commit fixes it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* vector worker: the drop guard follows the job's own commits

The guard watched the manifest of the version a job opened, and a job
that removes old versions itself, as a purge does, removed it and then
failed its next write as if the table had been dropped.

The guard now watches the newest manifest the job committed, and when
the job deletes the watched manifest itself, the newest manifest left
beside it. A table dropped under the job still fails its writes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* vector worker: a column add keeps requests and states committed meanwhile

add_columns commits the schema it read, and Lance lets that Merge go in
over the schema and field metadata updates committed while it ran. A key
requested, withdrawn or removed while another key's column was added was
silently undone: the removed key went back to ready and the request
vanished. The add now sets those again, and a key whose own request was
withdrawn meanwhile gets its new column off instead of being built.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* vector: keep FilterStats::record's doc on record

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* vector worker: test the guard against the job's own cleanup directly

The purge test asserted the purge was served, which depends on the
cleanup's retention policy rather than on the guard. It now has the
job remove every version before its own commit, the opened one with
them, through the guarded store, and write again; a purge after that
still completes. Without the previous commit's fix the write after the
cleanup fails with the DropGuard error.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* vector: indexed-key tests compare keys, not only distances

assert_same accepted a different key at a rank whenever its distance
matched, so a lost row replaced by a duplicate passed. It now requires
the same keys, and lets two keys trade ranks only when they tie.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* vector: filter stats record their table with each query

A query recorded its stats and its table in two steps, so the first
query of an index left the stats without a table, and a later table
under the name then took them over. A query of the old table finishing
after the index was created again also cleared or added to the new
table's stats.

record now takes the table and, under one lock, starts the stats over
for a new table, keeps the tables the name held before, and drops a
query of one of those.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* lance worker: asking again for a key change is done, not an error

UpdateMetadataKey commits the request and then lists the keys. When the
listing failed after the commit, the caller saw an error, and its retry
was refused as already indexed, so the change it made could not be
confirmed.

A request for a key requested or indexed already, and a removal of a key
not indexed or already off, now succeed without another commit. A request
for a key still being removed is refused.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* lance worker: a resumed add sets again what its commit overwrote

add_columns commits the schema it read, so add_column sets again the
requests and key states committed while it ran. A job that stopped
between the add's commit and that step left them overwritten, and the
build that resumed the key, seeing it building, finished it: a removal
asked for meanwhile was lost and the key went ready.

The added column records the version its add read. A resumed build
compares that version's schema with the add's and sets again what the
add overwrote before going on, and stops when that leaves the key off.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* vector: a key goes ready only after writes that raced its add are done

A PutVectors that opened the table before a key's column add and
committed after it wrote its rows' key column as of the version it read:
NULL for new rows, the old value for updated ones. It derives them again
itself, but when that failed, or the write was still running when the
build finished, the key went ready with those rows wrong, and maintenance
only filled NULLs.

PutVectors now commits within WRITE_DEADLINE (120s) of opening the table
or fails. finish waits until RACE_GRACE (180s) after the add, then derives
again the fragments written since it first looked, comparing values, not
only NULLs, before it marks the key ready.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* lance worker: drop guard tests for a table created again and flaky probes

A job on a table dropped and created again under the name, a probe that
fails after a manifest is written, a multipart upload completing after
the drop, and a cleanup whose paths now hold the new table's files. Each
fails against the guard as it is.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* lance worker: drop guard tells a table created again from the one opened

The guard only checked that the watched manifest path existed. A table
created again under the name reuses version paths, so a job on the old
table wrote into the new one. A failed HEAD after a manifest write was
taken as a drop and deleted the published manifest. A multipart upload
was checked only when it started, and a cleanup deleted paths it picked
before the drop.

The guard now records the manifest's e-tag and treats another e-tag, or
the manifest gone with others beside it, as a table created again. A
failed probe after a manifest write keeps the manifest. A multipart
upload checks again before it completes, and a cleanup checks once
before it deletes, refusing when another table is there.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* lance worker: order the zero-grace control of the racing-write test

The control ran the stale write 1.5s into finish, so on a slow runner it
could commit before finish's last pass and be repaired, failing the
assertion that it lands after ready. It now commits after finish returns.
With a grace the write still overlaps the wait: it commits 1.5s after the
add, and finish looks again only once the grace from the add has passed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* lance worker: set again what an add overwrote once, over later changes

recover_add replayed the add's historical delta on every resumed build,
so a request withdrawn after it was set again came back. It wrote over
values changed since the add, failed on a column dropped since, and
failed on every attempt once a cleanup removed the version it read.

The added column now carries a restore mark. The restoration is one
UpdateConfig commit that sets the schema entries and key states the add
overwrote and clears the mark, so it runs once. An entry or state
changed again since the add keeps its later value, and a column gone or
holding another key is skipped. The schema the add read comes from the
add's own version; without the version before it, the mark is cleared
and the build goes on. finish waits the whole grace from now when the
add's version is gone.

a_restored_add_is_not_replayed, a_resumed_add_skips_a_column_dropped_since
and a_resumed_add_goes_on_after_a_cleanup failed before this change.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* lance worker: a removal sets again what the key's add overwrote first

Recovery ran only when an add resumed. A key withdrawn after its add
stopped went to the remove job, which dropped the column and its restore
mark, losing for good the requests and key states the add overwrote.
The remove job now recovers the add before it drops the column; the key
stays off.

a_removal_keeps_what_the_add_overwrote failed before this change.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* lance worker: drop guard aborts on a failed probe, keeps another's manifest

An upload whose check before completing failed returned the error without
aborting, leaving its parts behind on S3. A manifest write found dropped
or replaced was deleted at its path even when a table created again had
since written its own manifest there.

complete now aborts on a failed probe too. The rollback deletes the
manifest only while its e-tag is the one this write got. Object stores
offer no conditional delete, so a table created again between that HEAD
and the delete is still not covered.

a_failed_probe_aborts_an_upload and
a_rollback_keeps_the_manifest_of_a_table_created_again failed before this
change.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* lance worker: test a failed rollback probe over a table created again

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* lance worker: keep a replacement's manifest when the rollback cannot tell it apart

After a write's manifest, a drop guard that found the table replaced
deleted the manifest whenever its HEAD failed or the PUT returned no
e-tag. It now deletes a replaced table's manifest only when the e-tags
match; a table merely dropped still takes its manifest back.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* vector: MetadataKeys leases the version it lists

Like the other reads, it pins the opened version so a purge cannot
reclaim that manifest's index files while the segments are loaded.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* lance worker: test a key that went ready on an empty table

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* lance worker: build the first segment of a ready key below the threshold

A key that went ready on an empty table has no segment. Detection held
its unindexed rows to min_unindexed_rows, so a small table stayed
unindexed. A ready key without a segment is now maintained once it has
rows, as index_fragments builds a first segment at any size.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* lance worker: test a repair retried over a compaction of its fragments

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* lance worker: a repair retried after contention derives the moved rows too

rederive started over from the latest version with the fragment ids it
was given. When a compaction caused the contention, those fragments were
gone and the retry repaired nothing, so maintain folded the compacted
rows with their columns still NULL. A retry now also derives the
fragments the newer version added.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* lance worker: test withdrawals that race a key's add

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* lance worker: a withdrawal that races the key's add sets its column off

A removal that read only the request cleared it, and Lance rebases that
metadata update over the add's Merge, so the column it made stayed
building and finish made it ready. After withdrawing the request, the
removal looks again and sets a column it finds for the key off.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* lance worker: a withdrawal recovers interrupted adds first

An add whose commit overwrote another key's request and stopped before
setting it again hides that request. A withdrawal of it then found
nothing to do, and the add's later recovery brought the request back.
request_removal now recovers such adds before it decides.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* lance worker: test a request committed while a column is dropped

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* lance worker: a drop sets again what it overwrote while it ran

drop_columns commits a Project of the schema it read, and Lance rebases
it over metadata updates, so a request or key state committed while the
column was dropped was lost. remove_column now sets those again as a
column add does, through the overwritten diff split out of
reapply_metadata.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* lance worker: drop test counts HEADs with try_update

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* vector: test a late query of a dropped table after the next table's first one

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* vector: test PutVectors at a version from before a key's column add

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* vector: keep a dropped table retired when its index's stats are forgotten

Dropping or creating an index removed its stats entry, and the retired tables with it, so a late query of the dropped table that came after the new table's first one reset the new table's stats.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* vector: refuse PutVectors at a table version

Table.version pins reads. A write opened at a version from before a key's column add would commit rows without the column after the key went ready, past the write deadline and the grace pass.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* admin: indexed metadata keys panel for S3 Vectors indexes (#12821)

* vector: report an index's rows and vector index type with its keys

MetadataKeys also answers with the table's rows and its vector index type
(IVF_FLAT, or empty without one), which the admin's index panel shows
next to the keys.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* admin: indexed metadata keys panel on the S3 Vectors bucket page

Each index of a vector bucket gets a card: dimension, metric, vectors and
vector index type; its indexed metadata keys with state, coverage of the
table's rows, filtered queries per hour and index size, with Remove for a
ready key and Cancel for a requested or building one; and the keys its
queries filtered on most in the last hour that are not indexed, with
their average latency. A key filtered only with numbers is marked
"(numeric: phase 2)" and has no Index button.

"+ Index a key" and a listed key's Index button open a dialog for the key,
which refuses the index's non-filterable keys. Adding or removing goes
through the engine; the admin then dispatches the lance_metadata_index
job for that key when a worker can run it, and detection finds it
otherwise. Filter stats are read from every registered engine and summed
per hour over each engine's window.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* admin: keep a metadata key's spaces when indexing it

The dialog trimmed the key before sending it, but metadata key names are
exact strings: a key with surrounding spaces would have indexed another
field. Only an empty or all-space name is refused now.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* admin: bound the index panel's key lookup

An engine that accepted the panel's MetadataKeys call and stopped
answering held the whole bucket page; the lookup now gives up after five
seconds and the panel shows the error, as the filter-stat calls do.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* admin: dispatch a metadata key job without the cluster admin lock

ExecutePluginJob holds the cluster admin lock for the whole run, so a
key build dispatched from the bucket page kept weed shell and the
default lane locked out for up to an hour. The Lance lane runs without
that lock in the scheduler; run the dispatched job the same way.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* admin: pass an index without non-filterable keys as an empty list

An index whose record holds a metadata configuration with no key list
was rendered as `null`, and the dialog's non-filterable check then threw
before it could submit or show an error.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* admin: ask in a modal before removing or cancelling a key's index

Remove and Cancel on a metadata key asked with the browser's confirm(),
unlike the rest of the page. They now open a Bootstrap modal laid out as
Delete Index's, naming the key and the index, and show a failure in the
modal.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* admin: show when an engine's filter stats leave keys out

sumFilterStats dropped each engine's truncated flag, so a panel whose
engine stopped counting new keys listed its frequent keys as complete.
The panel now carries the flag and says the list is partial.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-20 12:45:37 -08:00
Chris Lu 4c92a00357 admin/plugin: detect again soon after a capacity give-up (#12825)
A scheduled run whose jobs no executor can take gives them up at once,
leaving them to the next detection. For a job type with a long detection
interval that is a whole interval later: a daily job whose only worker
was briefly full lost a day.

Bring the job type's next detection forward to within 10 minutes when
its jobs are given up for capacity, never pushing back an earlier one.

Claude-Session: https://claude.ai/code/session_011UpvafcfUuC2mLxNAXLQJi
2026-11-20 00:31:18 -08:00
Chris LuandClaude Opus 5.5 1d7125abf2 filer: keep AppendToEntry's cluster lock for the whole append (#12819)
* filer: renew AppendToEntry's cluster lock while the append runs

AppendToEntry took a 5 s short-lived lock and never renewed it, so an
append slower than that (slow store, slow volume, GC pause) lost the
lock while still running, and another filer could take it and append
to the same entry concurrently, dropping one append's chunks. Use the
auto-renewing lock the HTTP append path already uses.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* filer: refuse an AppendToEntry commit once its cluster lock was lost

A renewal can still fail, or the lock can be lost and granted again
under a new generation, while the append runs; another filer may have
appended in between. Check the lock and its generation right before
the entry is written, fail the append so the caller retries, and drop
the manifest chunks this attempt created.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* cluster: IsLocked reports a lock whose lease ran out as not held

isLocked only flipped when a renewal came back rejected, a second after
the reply, so a holder whose renewal stalled past the lease (a long GC
pause) still read as locked while the server could hand the key to
someone else. Track the lease from each grant's send time and require
it in IsLocked, which the filer append paths check before committing.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* filer: hold the slow-append test past the renewing lock's first lease

The renewing lock grants a 7 s lease, so a 6 s hold passed even with
renewal off; wait past LiveLockTTL so only a renewal keeps the lock.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* filer: say the AppendToEntry lock check is best effort

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-20 00:01:24 -08:00
Chris LuandClaude Opus 5.5 6018f87c25 test(s3vectors): add an author-filter workload to the perf suite (#12818)
* test(s3vectors): let a perf test restart the engine with extra environment

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* test(s3vectors): serve the volume server's metrics in the test cluster

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* test(s3vectors): build a named index in the perf suite through one helper

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* test(s3vectors): add an author-filter workload to the perf suite

Plants authors at 10%, 1%, 0.01%, fewer than topK and no rows (seed 5),
scattered over the index or written as consecutive keys, and reports the
eligible rows, p50/p95, filtered recall@k against an exact numpy search
over the eligible rows, results per query, hits without a finite
distance, and the engine and volume server read counters, warm or with
the engine's caches at 0 MB.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* test(s3vectors): reject duplicate hits and clean up a failed filter workload setup

The workload skips the distributed cluster, since only one engine is
restarted, and accepts only the warm and cold modes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-19 23:48:58 -08:00
Chris LuandClaude Opus 5.5 4bbc03747f s3vectors: match json_extract's escaping of control characters in filter strings (#12817)
* s3vectors: render filter string control characters as serde_json does

json_extract serializes the stored value with serde_json: \b, \f, \n,
\r, \t, \u00XX for the rest of U+0000-U+001F, and U+007F-U+009F raw.
The scalar comparison text used Rust debug escapes (\0, \u{1}, \u{7f}),
so $eq/$in on such strings missed scalar values and $ne/$nin kept them.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* test(s3vectors): filter on every control character, quote and backslash

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-19 23:48:27 -08:00
Chris LuandClaude Opus 5.5 e109f91c6f vector: indexed metadata keys, engine side (derive, rewrite, exact planner, filter counter) (#12813)
* vector: derive the indexed metadata key columns

An indexed metadata key is a hidden `_mk_<n>` column of List<Utf8>: the
strings an S3 `$eq` on a string matches, that is the value when it is a
string or the string elements of an array, and nothing for any other type
or a missing key. The field metadata names the key and its state
(`building` or `ready`).

The module derives the column from metadata given as JSON (the write
path) or as stored (jsonb or JSON text, for a backfill), and reads the
indexed keys from a schema. It lives in the worker crate so the engine and
the worker share it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* vector: fill the indexed metadata key columns on PutVectors

PutVectors builds its batch against the table's current schema. Once a
table has indexed metadata key columns, the three-column batch no longer
matches that schema and the write fails. Fill each `_mk_*` column the
schema has from the row's metadata, whatever the key's state; a column
the engine does not know still fails the request.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* vector: answer string $eq and $in on a ready indexed key from its column

The gateway renders a string `$eq` or `$in` on a key as an OR of
`json_extract(metadata, P) = '"v"'` (or IN) and
`json_array_contains(metadata, P, '"v"')` over the same values. On the
parsed filter, a top-level conjunct of that shape on a key whose column is
`ready` becomes `array_has` / `array_has_any` on the column, which its
LABEL_LIST index answers. The two halves must name the same key and the
same strings, the scalar half in the text json_extract returns for them;
anything else, including `$ne`, `$nin`, `$exists`, numbers and booleans,
stays on the JSON path, and the other conjuncts still apply.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* vector: score few indexed matches exactly, per scope

With a ready indexed key in the filter, `ann` first counts the rows of its
scope that the key's conjuncts match, through the scalar index. When they
are at most the limit it scores them exactly over that scope, the way
the short-result fallback does; above it, the vector index searches with
the indexed prefilter, and the fallback still applies. The scope is the
one `exact_in_scope` scores, so per-scope exact answers merge into the
global exact top-k.

The limit is `Query::exact_match_limit` (None takes 256; 0 never counts),
set by the engine's --exact-match-limit (WEED_VECTOR_EXACT_MATCH_LIMIT).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* vector: count filtered queries by metadata key

For each index, the engine counts its QueryVectors calls by the metadata
keys of the parsed filter, with their total latency and the operand types
each key was compared with, over the last hour in 5-minute buckets. At most
512 indexes and 32 keys per index are kept (an idle key makes room;
otherwise the index is marked truncated), and the stats are not exported
as metrics labels.

The new FilterStats RPC returns an index's stats; the gateway's discovery
client forwards it like the other calls.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* vector: report the metadata columns capability in gRPC health

An engine whose writes fill the indexed metadata key columns reports the
gRPC health service `weed.vector.capability.metadata_columns` as serving,
so an operator action can refuse to index a key while an older engine
could still write the table.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* vector: expire filter stats' operand kinds and truncation with the window

The operand kinds a key was compared with, and the mark that a key went
uncounted, outlived the queries that set them. Keep the kinds per
5-minute bucket and the truncation as the bucket it last happened in, so
both leave the hour window with the counts.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* vector: fill indexed key columns a write left empty racing their add

A PutVectors that opened the table before an indexed key's column was
added, and commits after the add, is rebased by Lance onto the wider
schema: it commits, and the rows it wrote hold NULL in the new column, so
the column's index misses them (reproduced in the test: 100 of 150 rows).

After its commit, PutVectors now looks for indexed-key columns the batch
did not carry, reads those of its keys whose column is still NULL, derives
the values from their stored metadata and patches just those columns with
a partial-schema merge that updates only rows still NULL.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* vector: keep filter stats' truncation mark at the later bucket

record() reads the clock before it takes the lock, so a call that waited
on the lock across a bucket boundary could move the truncation mark (and
the index's last-used bucket) back to its earlier bucket, and the mark
then left the window one bucket early.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* vector: derive a raced write's indexed key columns from current metadata

A PutVectors that opened the table before an indexed key's column was
added is retried by Lance against the wider schema: rows it inserts hold
NULL there, and rows it updates keep the value derived from their old
metadata, which the NULL-only heal never looked at. The heal now derives
the columns for all of the write's keys.

Its merge also ran with Lance's retry, which joins the values derived
before the conflict onto the rows a later write of the same keys left.
It now fails on a conflict and is redone from the latest version.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* vector: check the raced write's keys keep the later write's rows

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-19 23:45:30 -08:00
Chris LuandClaude Opus 5.5 817338c911 volume.merge, volume tail receiver: apply size-0 records for existing keys as deletes (#12816)
* volume: apply a tailed size-0 record as a delete when the key exists

A delete and an empty write both reach the tail receiver as a size-0
.dat record. Writing every one as a live needle turned each delete
caught by volume.move's tail into a 200-empty object. Treat the record
as a delete when the key already has a record on the receiver, and as
an empty write otherwise.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* rust volume: apply a tailed size-0 record as a delete when the key exists

Mirrors the Go tail receiver: a size-0 record for a key that already
has a record here is a delete, otherwise an empty write.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* shell: volume.merge applies a size-0 record as a delete when the key exists

The tail stream carries deletes as size-0 records, and volume.merge
wrote each one to the merged copy as a live empty needle, so every
delete in the volume came back as a 200-empty object on all replicas.
A size-0 record for a key already written in the merged stream is now
a delete on the target; for a new key it stays an empty write.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* shell: volume.merge reads a size-0 record as a delete only after its own replica's record

Replicas append the same write at their own times, so a fresh empty write
can reach the merge twice when the copies are further apart than the
dedup window. Tracking keys across all replicas made the second copy a
delete. Track, per source replica, the keys it has returned (duplicates
the merge skips included): a size-0 record is a delete only when its own
replica wrote the key before, which is the only way a replica appends a
delete. A delete for a key the merged copy already deleted sends nothing.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* shell: volume.merge verification counts the deletes the merge applied

Applying deletes lets a correct merge hold fewer live needles than a
replica that missed a delete, and the live-count check then refused it
and kept the divergent replicas. Every key live on a replica is either
live or deleted in the merged copy, so compare that replica's live count
with the merged live count plus the keys the merge deleted. An empty
merged copy is still refused.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* volume: tail receiver looks a size-0 record's key up in the needle map

The receiver read the needle to decide whether a size-0 record deletes
an existing key. A read reports a needle past its TTL as not found, so
a delete for it was written as a live empty needle, which reads ignore
TTL for. Look the key up in the needle map instead (Volume.LookupNeedle).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* rust volume: tail receiver looks a size-0 record's key up in the needle map

Mirror Go: write_tailed_needle read the needle to decide whether a size-0
record deletes an existing key, and a read reports a needle past its TTL
as not found, so the delete became a live empty needle. Look the key up
in the needle map instead (Volume::lookup_needle).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* shell: volume.merge keeps a merged delete when a late empty copy follows

A replica that missed a delete can hold its copy of the empty write
stamped after the other replica's delete. As that replica's first record
for the key it was written, reviving the deleted key. A size-0 record now
never revives a key the merged copy already deleted.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* shell: volume.merge refuses more replicas than its key history has bits

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-19 22:36:44 -08:00
0a1e337302 wdclient: keep a volume's live replicas readable while another volume cannot be found (#12814)
* wdclient: keep a volume's live replicas readable while another volume cannot be found

A filer's "volume not found" answer counted as a filer failure, so three
of them opened the circuit breaker on a healthy filer. With its only filer
in the cool-down, every lookup then failed without asking, as "all 1
filer(s) failed with non-retryable error: %!w(<nil>)". And a refresh whose
lookup failed left the cache without the volume's live replicas, since
InvalidateCache had already dropped them with the dead one.

The breaker now counts only failures to reach the filer, and asks the
filers anyway when all of them are in their cool-down. InvalidateCache keeps
what it removed; a lookup that errors puts it back unless the volume was
learned again, the epoch moved, or a reset would have expired it, and
leaves out locations fenced by a pushed removal.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* wdclient: set invalidated locations aside in the map instead of copying them out

The first commit kept what InvalidateCache removed in a side map and put it
back when the lookup failed. The copy left the map's lock, so a removal the
master pushed between the take and the put-back was undone: the removed
server came back into the cache.

The entry now stays in the map with a state. entryStale hides it from readers
and releases its server references, as a delete did. A pushed removal edits
it like any other entry. When the lookup after the invalidation returns an
error, the entry becomes entryRevived and is served again.

A stale or revived entry holds only what was known before the read failed,
so the next thing learned replaces it: a lookup answer drops it before the
answer is cached, including an answer with no locations and an answer to a
lookup that was in flight when another one failed, and a pushed add replaces
it instead of merging.

LookupVolumeIdsWithFallback returns no error when every volume it was asked
for has locations after the revive, so the filer's LookupVolume handler sends
its response instead of an error status.

The not-found breaker test now checks the failure count itself. Without
that, a regression of the classification passed, because a lone filer in its
cool-down is asked anyway.

On the first commit, TestPushedRemovalRacingFailedRefreshes failed in 8 of 8
runs under -race: four with "got [live:8080 old:8080], want [live:8080]",
four with "got [], want [live:8080]".

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>

* wdclient: retry a filer lookup pass that asked no filer

The pass decides once whether to skip unhealthy filers. A filer whose
breaker trips after that decision, as when a concurrent lookup records
its failure, is still skipped. With every filer skipped the pass ended
with no error and failed the lookup as non-retryable with
"%!w(<nil>)". Such a pass now counts as Unavailable, so the lookup
retries and the next pass asks the filers anyway.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* wdclient: believe a provider answer that reports errors for some volumes

The master provider answers a batch with a map of the volumes it knows
and an error naming the ones it does not. The failed-lookup branch took
that as no answer and revived what InvalidateCache had set aside, so a
volume the master reported as absent came back with its old locations,
and the batch could return them with a nil error. Only a nil map now
means no answer. Otherwise the answer replaces what was set aside, and
the provider's error is still returned.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* wdclient: keep set-aside locations learned after a lookup was asked

A successful lookup dropped every stale or revived entry of the volumes
it asked for, however old the answer was. A lookup asked before the
master confirmed new locations could, on finishing, delete those
locations after a read had set them aside and a failed refresh had
revived them, and its own answer was not cached because the epoch had
moved. Entries now record when their locations were learned, and an
answer drops only what was learned before it was asked.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* wdclient: keep an older answer from replacing locations set aside after it was asked

fdcde25 keeps dropSetAside from deleting set-aside locations that were
learned after a lookup was asked. The answer's insertion still replaced
them: fillLocations adds each location with vidMap.addLocation, and
addLocationToMap replaces a stale or revived entry whatever its learnedAt.
The epoch does not stop this when the newer locations came from another
lookup's answer or, on the master client, from a pushed add, since neither
moves it. The master client's provider also caches its answer itself,
before LookupVolumeIdsWithFallback reaches dropSetAside.

vidMap.addAnswer adds an answer's locations under one lock, and adds none
of them while vid has a stale or revived entry learned after the lookup was
asked. fillLocations uses it, and the master client's provider passes the
learned mark it takes next to its epoch. An answer with nothing newer in the
way still replaces what was set aside, with every location it reports.

On fdcde25, TestOlderAnswerDoesNotReplaceNewerRevivedLocations and the
"newer location revived during the lookup" case of
TestMasterAnswerAgainstSetAsideLocations fail with
"got [old:8080], want [new:8080]".

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* wdclient: move addAnswer next to dropSetAside

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
2026-11-19 22:34:35 -08:00
Chris LuandClaude Opus 5.5 374c77bcbe admin/plugin: fail a restored job its worker no longer lists (#12815)
* admin/plugin: fail a restored job its worker no longer lists

An admin restart reloads the jobs that were running from the persisted
state, but the workers drop those jobs when the old admin stream closes.
No pending execution covers a restored job, so only the 24h stale sweep
ended it, and until then it blocked dedupe and scheduling of its job type.

Track the active jobs loaded at startup by worker and fail one after two
itemized heartbeats from its worker leave it out, as is already done for
a dispatch whose worker restarted. A listed job is left to adoption.

Claude-Session: https://claude.ai/code/session_011UpvafcfUuC2mLxNAXLQJi

* admin/plugin: keep tracking a listed restored job; skip queued ones

- A restored job its worker lists only resets its miss count, so one it
  drops later is still failed.
- Track only jobs that were sent to a worker (request id set, not pending).
- Match restored ids against the trimmed heartbeat ids.
- Fail the restored attempt under the jobs lock that applies the expiry
  (handleJobCompletedIf / expireActiveJobIf), so a newer attempt of the
  same job is neither failed nor canceled.

Claude-Session: https://claude.ai/code/session_011UpvafcfUuC2mLxNAXLQJi

* admin/plugin: keep a newer attempt's detail from a restored-job expiry

- The conditional completion writes the job detail through
  persistJobDetailSnapshotIf and skips it when the stored detail already
  names another request; the check runs under jobDetailsMu with the write.
- mergeTerminalDetailIntoTracked ignores a terminal detail of another
  request, so a restart does not end the loaded attempt with it.
- Tests wait for the restored-job expiries instead of sleeping.

Claude-Session: https://claude.ai/code/session_011UpvafcfUuC2mLxNAXLQJi

* admin/plugin: restore a finished retry from its detail

The job detail is saved as each attempt changes and the job list only
periodically, so a terminal detail naming another request is the newer
attempt. Apply it with its request id instead of ignoring it; the
expiry's own detail write already refuses to overwrite another request.

Claude-Session: https://claude.ai/code/session_011UpvafcfUuC2mLxNAXLQJi

* admin/plugin: restore whichever attempt is newer

The job list and a job's detail can name different attempts either way
round: the list is saved periodically, and a detail write can fail. When
their request ids differ, apply the terminal detail only if it is the
newer attempt (higher attempt number, else later update).

Claude-Session: https://claude.ai/code/session_011UpvafcfUuC2mLxNAXLQJi

* admin/plugin: fill a restored job's empty request id from its detail

A job the list saved while still queued has no request id; take the
finished detail's, as the merge did before the newer-attempt check.

Claude-Session: https://claude.ai/code/session_011UpvafcfUuC2mLxNAXLQJi

* admin/plugin: take the worker and attempt of a newer restored detail

When the restore takes a finished retry's detail over the job list's
attempt, the record kept the older attempt's worker and attempt number
next to the retry's request id and state.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* admin/plugin: fold the conditional completion into the existing functions

handleJobCompleted, expireActiveJob, trackExecutionCompletion and
persistJobDetailSnapshot take the accept check themselves instead of
gaining *If twins.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* admin/plugin: ignore a late earlier-attempt completion; replace an older detail

- A completion whose request id differs from the tracked attempt's is a
  late report of an earlier attempt and no longer ends a retry.
- A conditional completion skips the detail write only when the stored
  detail is a newer attempt (detailIsNewerAttempt), so an older finished
  attempt's detail left behind by a failed write is replaced.

Claude-Session: https://claude.ai/code/session_011UpvafcfUuC2mLxNAXLQJi

* admin/plugin: take only the dispatched attempt's reports

A late progress report of a timed-out attempt replaced the retry's request
id, after which the retry's own completion was rejected as foreign. One
rule now gates both: a record the admin dispatched (attempt > 0) takes
progress and completion only for its own request (reportFromOtherAttempt).

When an expiry replaces an older attempt's detail it also takes the
closing attempt's worker, attempt number and times.

Claude-Session: https://claude.ai/code/session_011UpvafcfUuC2mLxNAXLQJi

* admin/plugin: restore a newer running attempt's identity from its detail

The job list can name attempt A while the detail shows retry B running
when the admin stopped. The restore now takes B's request id, worker and
attempt number from such a newer running detail, so B's own reports are
accepted. Test helper keeps a given attempt number.

Claude-Session: https://claude.ai/code/session_011UpvafcfUuC2mLxNAXLQJi

* admin/plugin: keep attemptEnded's doc comment on attemptEnded

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* admin/plugin: dispatch an attempt-less job as attempt 1

The late-report guard only covers records with a positive attempt, and
ExecuteJob passed a zero attempt through, so a direct caller's dispatch
could be closed by another request's completion. The HTTP API already
raised zero to one; the dispatch path now does the same for every caller.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-19 22:30:52 -08:00
2382f07788 filer: keep a hard-linked file's chunks when one of its names is replaced (#12812)
* filer: keep a hard-linked file's chunks when one of its names is replaced

Replacing one name of a hard-linked file deleted the file's chunks while
its other names still pointed at them. Reads through those names then
return EIO once the volumes are vacuumed; until then they only work
through readDeleted.

Two everyday ways to get there through a mount:
- conda links an environment's files to its package cache with hard
  links. Python saves a __pycache__/*.pyc by writing a temporary file and
  renaming it over the old name, so the cache's copy loses its data.
- A training script saves model.pt, hard-links it into an epoch directory
  and saves the next epoch by renaming a new file over model.pt. Each
  archived epoch loses its data.

Three write paths deleted the chunks of the entry they replaced without
asking whether that entry was one of several hard links:
- a rename over a file: moveSelfEntry drops the target's link, then
  queues all of the target's chunks for deletion;
- CreateEntry over an existing path (an S3 PUT, for one): the update
  drops the old entry's link, then deleteChunksIfNotNew deletes its chunks,
  and the CreateEntry handler deletes chunks its own list covers;
- UpdateEntry with an entry that no longer carries the old one's link.

A plain delete already keeps the chunks while HardLinkCounter > 1, and a
folder delete asks hardLinkIsOrphaned. The three paths now go through
Filer.KeepHardLinkedChunks, which asks after the write, once the replaced
name's link is gone: the chunks stay unless the file's shared record went
with it, that is, unless the replaced name was the last link. Writing the
file through one of its names (an entry with the same hard link id) still
deletes the chunks the write replaced, since every name reads the new
content.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* filer: keep hard-linked chunks from the shared record, not the replaced row

A lookup that cannot read the hard link record returns the name's own row,
which can lag the file. Read the record when filtering the garbage, and keep
everything when it cannot be read.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* filer: test the manifest reads behind KeepHardLinkedChunks

The manifest resolution moves into withoutChunksOf, which takes the lookup
function, so tests can serve manifests. Chunks behind nested manifests and
the manifests themselves stay, unrelated garbage is still deleted, and an
unreadable manifest returns an error, on which KeepHardLinkedChunks deletes
nothing. Behavior is unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
2026-11-19 22:14:01 -08:00
Chris LuandClaude Opus 5.5 a91def90c0 volume: count and log needles vacuum drops as unreadable (#12811)
* volume: count needles vacuum drops as unreadable and log each at error level

Vacuum drops a live needle whose record fails to read back intact (CRC
mismatch, malformed fields, short or dangling record). The key leaves the
compacted index with no tombstone, so repair tools later treat it as a
vacuumed delete. Log every dropped key at error level and count the drops
in SeaweedFS_volumeServer_vacuuming_dropped_needles per collection, once
the compacted index is written.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* grafana: add a panel for needles dropped by vacuum

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* seaweed-volume: count needles vacuum drops as unreadable and log each at error level

Mirror the Go volume server: log every dropped key at error level and add
SeaweedFS_volumeServer_vacuuming_dropped_needles per collection once the
compacted index is written.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* volume: count dropped needles only after the compacted dat is flushed

The final .cpd fsync runs in the deferred close and loadTmpNeedleMap can
still fail after the index is saved; a copy failing there is retried and
would count the same needles again.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-19 21:02:31 -08:00
Chris LuandClaude Opus 5.5 2ac8db5cb4 filer: lock the path for plain HTTP overwrites so replaced chunks get deleted (#12801)
* filer: lock the path for plain HTTP overwrites

CreateEntry looks up the entry it replaces and queues that entry's chunks
for deletion. Two plain PUT/POST writes of one path could both read the
same old entry, so the chunks of whichever write lost were never queued.
Take the exclusive path lock around CreateEntry, as the gRPC CreateEntry
already does.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* filer: test concurrent HTTP overwrites queue every replaced chunk

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-19 21:01:08 -08:00
Chris LuandClaude Opus 5.5 ac7d949c61 filer: keep a manifest needle until its data chunks are queued for deletion (#12810)
* filer: resolve deleted manifests through the master on a volume cache miss

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* filer: keep a manifest needle until its sub-chunks are queued for deletion

When a manifest could not be read during chunk deletion, the manifest
needle was deleted anyway and the data chunks it listed were orphaned.
Retry the resolve with the deletion retry backoff and delete the
manifest alone only after MaxRetryAttempts.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* filer: keep a durable delete that joins a pending manifest retry

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* filer: bound each manifest retry read with a deadline

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* filer: requeue resolved manifest ids on an fsync upgrade instead of rereading

A durable delete that joins a manifest retry in flight used to resolve
the manifest a second time, but the first pass had already queued the
manifest needle, so the reread could fail and the sub-chunks never got
the durable mark. Resolve nested manifests up front and queue the same
ids again with fsync.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* filer: bound each manifest retry pass and stop it on shutdown

A pass takes ready manifests one at a time and stops after
manifestRetryTimeout, leaving the rest queued for the next tick, so a run
of stalled reads cannot keep the shared retry loop from ordinary deletion
retries or from noticing shutdown.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-19 19:50:40 -08:00
Chris LuandClaude Opus 5.5 8df716b728 volume: make a delete retry after a failed replica fan-out reach the replicas (#12797)
* http: report the status of a failed delete that has no body

A replica that answers a DELETE with a bare 500 left the forwarding
server reporting "[<replica>]: " with an empty reason.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* volume: forward a delete retry that finds nothing here to the replicas

A delete is applied locally before it is sent to the other replicas, so
when the fan-out fails the client is told to retry while this replica
already dropped the needle. The retry then read nothing here and answered
404 without asking the replicas, so the replica that missed the delete
kept the needle for good.

On a replicated volume, a plain delete that finds the needle missing or
deleted now cancels a write still queued for it, as a replica delete
does, and sends the delete on to the other replicas. A failed fan-out is
reported as before, so the client retries until every replica agrees.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* volume(rs): take a replica's 404 to a forwarded delete as done

A replica that no longer holds the needle answers a forwarded delete with
404. The Go server folds that into success, but the Rust server reported
it as a replication failure, so a delete that only had to catch up one
replica failed on the replica that was already in step.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* volume(rs): forward a delete retry that finds nothing here to the replicas

Mirrors the Go server: on a replicated volume, a plain delete that finds
the needle missing or deleted cancels a write still queued for it and
sends the delete on to the other replicas, so a retry after a failed
fan-out reaches the replica that kept the needle.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* volume: answer a delete whose needle read fails with 500, not 404

A replica's 404 to a forwarded delete is taken as the needle being gone
there. DeleteHandler answered 404 for any failed pre-read, so a replica
whose disk could not read the needle let the primary acknowledge a delete
that replica never applied. Only a missing or deleted needle (or a volume
this server does not hold) answers 404 now; other read errors are a 500,
so the client retries.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* volume(rs): answer a delete whose needle read fails with 500, not 404

Mirrors the Go server: a replica's 404 to a forwarded delete is taken as
the needle being gone there, so only a missing or deleted needle answers
404; other read errors are a 500 and the client retries.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* volume(rs): corrupt the needle with portable file writes in the delete read test

FileExt::write_all_at is Unix-only, so the test did not build on Windows.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-19 19:47:02 -08:00
Carlos Leyva 2e4be5e25e mount: report a read-only mount root that does not exist as not found, not as the create error (#12799) 2026-11-19 19:45:41 -08:00
Chris LuandClaude Opus 5.5 f2acfb7be4 volume: serve decoded bytes for a range over a compressed needle (Go and Rust) (#12805)
* volume: serve decoded bytes for a range over a compressed needle

A Range with Accept-Encoding: gzip got Content-Encoding: gzip and a slice
of the compressed bytes, which a decoding client cannot read. The range
is over the stored file's decoded bytes, so pass gzip through only for a
whole read.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* seaweed-volume: serve decoded bytes for a range over a compressed needle

Mirrors the Go volume server: a Range with Accept-Encoding: gzip got
Content-Encoding: gzip and a slice of the compressed bytes. Pass gzip
through only for a whole read.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* seaweed-volume: HEAD of a compressed needle reports the served length

HEAD answered a compressed needle from its meta: the stored gzip length
with no Content-Encoding, whatever the request accepted. Go reads the
payload and reports what the GET would send: the decoded length, or the
gzip length with Content-Encoding: gzip for a whole read that accepts
gzip. Read a compressed needle in full on HEAD too.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* test/volume_server: range and HEAD over a compressed needle

Both volume servers serve a gzip-accepted range decoded, and HEAD reports
the length the matching GET would send.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* seaweed-volume: decode every gzip member of a compressed needle

Go's gzip reader reads concatenated members; GzDecoder stopped after the
first, so a range over such a needle saw a shorter body and length.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-19 19:43:02 -08:00
Chris LuandClaude Opus 5.5 b5f23b7dbf vector: answer a short filtered ANN query by an exact scan (#12802)
* vector: answer a short filtered ANN query by an exact scan

A prefiltered query through the IVF index sees only the partitions it
probes. When those hold fewer than k eligible rows the answer is short
(400 matches, topK 500, 63 partitions: about 208 returned), and when at
most k rows match Lance appends the ones its probes missed with an
infinite distance. Either case is now answered again by an exact scan of
the same scope: the listed fragments plus the live coverage of the listed
index segments, or the whole table.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* vector: count an explicit probe count as indexed only with an index

With nprobes set on a table that has no vector index, the scan was marked
indexed, so a filtered query matching fewer than k rows ran the same exact
scan a second time.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-19 19:42:02 -08:00
261c534c19 filer: retry TiKV store writes on a write conflict (#12798)
* filer: retry TiKV store writes on a write conflict

TiKV transactions are optimistic. A commit fails with a write conflict
when another transaction committed one of its keys after it began, even
for a blind Set or Delete. Outside a filer transaction the TiKV store
runs each save and delete in a transaction of its own and returned that
error to the caller, so of two requests writing the same entry at the
same moment one failed, and a mount reported EIO.

A write the store commits itself (InsertEntry, UpdateEntry, DeleteEntry,
one batch of DeleteFolderChildren, KvPut, KvDelete) now runs again in a
new transaction after a write conflict, up to 5 times, waiting 1-2ms,
2-4ms, ... 16-32ms in between. The failed transaction stored nothing,
and the retry lands after the other writer, as if the two had not
overlapped. The retries stop when the context is done.

A write inside a transaction from BeginTransaction is not retried: the
conflict is reported when the caller commits, as before.

deleteBatch now goes through RunInTxn like the other writes.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>

* filer: stop TiKV write retries once the context is done

When the context was cancelled while the retry wait was running, the
timer case could still win the select and start one more transaction.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* filer: keep a TiKV entry that is no longer expired when listing deletes it

The listing decides an entry has expired from its own read, and the
delete ran later as a blind delete. Once the delete retries after a
write conflict, a live replacement committed in between was removed.
Read the entry again in the deleting transaction and delete it only if
it is still expired.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: MapleWhiteZZZ <18653045+MapleWhiteZZZ@users.noreply.github.com>
Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
2026-11-19 19:17:26 -08:00
github-actions[bot] 5e003bae7d docs: regenerate star history chart 2026-11-20 01:22:33 +00:00
Chris LuandClaude Opus 5.5 4ac3633f1c volume: check the needle id in the streamed-read probe and on blob writes (#12794)
* storage: reject a streamed read whose record id or size differs

The header probe before a streamed read returned not-found only when both
the record's id and its size differed from the index entry, so a record of
another needle with the same size passed and its data was streamed under
the requested key until the final checksum check.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* storage: refuse a needle blob whose header names another needle

WriteNeedleBlob compared only the blob header's size with the size it was
asked to index. A blob read from an offset that has since moved to another
record of the same size was appended and indexed under the wrong key.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* volume(rs): refuse a needle blob whose header names another needle

Mirror of the Go check: write_needle_blob_and_index compared only the blob
header's size, so a blob of another needle with the same size was appended
and indexed under the requested key.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* volume(rs): report not-found when a streamed read's record is another needle

The meta read ahead of a streamed reply took the id from the record header
without comparing it with the requested key. Return not-found when the
index entry resolves to another needle, as the Go header probe does.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* storage: check the requested id after a meta-only read

ReadNeedleMeta overwrites n.Id with the record's header id, so a meta-only
read through an index entry that resolves to another needle returned that
needle's metadata, and the streamed-read probe that follows compared the
record with itself. Return not-found when the header names another needle,
as the Rust meta read does.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test: copy a needle blob to another volume, not under another key

WriteNeedleBlob now refuses a blob whose header names another needle,
so the round trip writes the blob under its own id into a second volume
and checks that storing it under another key is refused.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* storage: compare the streamed header with a deleted entry's record size

A deleted index entry stores the negated size, so a read-deleted stream
compared the positive header size with a negative one and missed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-19 15:27:15 -08:00
Chris LuandClaude Opus 5.5 4a5dbe4c6f storage: copy an empty needle written during compaction in makeupDiff (#12796)
* storage: copy an empty needle written during compaction in makeupDiff

A size-0 index row is a live empty needle, not a delete. makeupDiff
grouped it with tombstones and appended a made-up delete marker, so
the compacted .dat lost the record's cookie and held a delete.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* volume(rs): copy an empty needle written during compaction in makeup_diff

A size-0 index row is a live empty needle. makeup_diff wrote a made-up
delete marker and a tombstone row for it, so the needle read as not
found after the commit. Copy the record like any other live row.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* storage: fail makeupDiff when the replayed needle write fails

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* volume(rs): check the compacted empty needle's id and cookie in the test

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-19 15:26:20 -08:00
Chris LuandClaude Opus 5.5 ba1bfa816c admin: show a pruned job's detail from its run record (#12795)
* admin: show a pruned job's detail from its run record

Tracking keeps the newest 1000 jobs across all job types and deletes
the details of older finished ones, while each job type's run history
keeps its last 10 successful and 10 failed runs. A job listed in the
run history could therefore open as "job detail not found". Fall back
to the run record when the job is no longer tracked.

Claude-Session: https://claude.ai/code/session_011UpvafcfUuC2mLxNAXLQJi

* admin: show 100% progress for a pruned successful run

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* admin: use the newest run record for a job across both outcomes

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* admin: keep the run record a pruned job's detail was built from

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* admin: keep looking for a run record past an unreadable history

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* admin: pick a pruned job's newest run record across job types

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-19 15:23:46 -08:00
Ping QiuandClaude Opus 5.5 be8ebc2e04 storage: build the temporary needle map in CompactByVolumeData (#12788)
* storage: red test for CommitCompact after CompactByVolumeData with a concurrent write or delete

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* storage: build the temporary needle map in CompactByVolumeData

Fixes #12783

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* storage: require a successful commit in the CompactByVolumeData commit test

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-19 15:06:05 -08:00
Chris LuandClaude Opus 5.5 bfbeed61a7 storage: read only the header and trailer in the append-time search (#12791)
BinarySearchByAppendAtNs, its left/right probes and ResumeByLastEntry read
every probed record through readNeedleAt, which parsed the whole body and
verified the data checksum, although they use only the id, cookie, size and
append time. One record with damaged data on the search path, even a needle
deleted long ago, failed the search and with it the tail sender, a
timestamp-based incremental copy and the volume.move catch-up.

readNeedleAt now reads the header and the trailer and skips the data. The
data checksum is still verified where the data is read, including the tail
receiver for every record it gets. The Rust volume server already parses
only the metadata on this path and does not verify the data checksum.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-19 15:02:25 -08:00
05427e8108 storage: hold the data-file read lock across the append-time search (#12789)
* storage: red test for the append-time search racing a vacuum commit

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* storage: hold the data-file read lock across the append-time search

Fixes #12785

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* storage: Version() writes the superblock version only when it changes

Version() stored the .vif version into SuperBlock.Version on every call, so
each call was a write that raced with readers of the superblock that do not
take superBlockAccessLock, such as the append-time search and the
compaction setup.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* storage: harden the append-time search race test; add a lock-hold budget test

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* storage: Linux-only lock budget; stop before reading search failures

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
2026-11-19 15:01:47 -08:00
Chris LuandClaude Opus 5.5 f7323f2b63 filer: optional HTTP Basic Auth on the filer HTTP ports (#12778)
* security: add a constant-time HTTP Basic Auth check

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* filer: require HTTP Basic Auth on the HTTP ports when configured

With filer.basic_auth.password set in security.toml (or
WEED_FILER_BASIC_AUTH_PASSWORD), a request on the filer HTTP ports, TUS
included, must carry those credentials or a valid filer JWT, and an
access level with no JWT signing key is no longer open. Valid Basic
credentials grant read and write. 401 responses then carry a Basic
challenge so browsers prompt. The credentials reload on SIGHUP with the
signing keys.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* scaffold: document filer.basic_auth in security.toml

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* filer: enable reloaded Basic Auth before swapping the JWT keys

Reload cleared the old signing keys before storing the new Basic
credentials, so a switch from JWT to Basic Auth briefly left both off
and admitted anonymous requests. Store new credentials first and clear
old ones last.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* filer: refuse Basic credentials on cross-origin browser requests

Browsers resend cached Basic credentials on requests other websites
trigger, such as a form POST with mv.from, and the default
cors.allowed_origins "*" lets them through. Accept Basic credentials
only on requests without an Origin, from the filer's own host, or from
an origin explicitly listed in cors.allowed_origins. A JWT is still
accepted from any origin.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* filer: recheck Basic Auth after reading an empty signing key

A request that loaded Basic Auth as off, then read the signing key after
a JWT-to-Basic Reload had cleared it, was let through anonymously. Open
access now also needs Basic Auth to be off after the key read; with
Reload enabling the new credential before disabling the old one, the
two loads cannot both miss a credential that was on throughout.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* security: compare Basic Auth credentials as MACs under a per-process key

A plain SHA-256 of the password is what password-hash scanners flag;
keying the digests with a random per-process HMAC key keeps the
length-independent constant-time compare without holding one.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* util: read a configuration file without merging into the shared viper

LoadConfiguration merges each read into the shared viper, so a key
removed from the file keeps its old value after a reload.
ReadConfiguration reads the file into a fresh viper with the same search
path and WEED_ environment overrides.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* filer: turn Basic Auth off when security.toml no longer sets it

The shared viper merges each SIGHUP reload into what it held, so
removing [filer.basic_auth] kept the old password valid. Read the
credentials from security.toml alone at startup and on reload; an
unreadable file keeps the current setting.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* filer: keep Basic Auth when security.toml cannot be read

A malformed file at startup dropped a WEED_FILER_BASIC_AUTH_PASSWORD
from the environment, and viper reports an unsearchable config
directory as a missing file, which turned Basic Auth off on reload.
ReadConfiguration now always returns the environment overrides and
reports whether the file was read; a reload that could not read it
keeps the current credentials.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* filer: keep reloading Basic Auth from the security.toml first read

A search directory that turned unsearchable made viper fall through to a
lower-priority security.toml, and a reload from a file without
[filer.basic_auth] turned Basic Auth off. ReadConfiguration now takes the
file to read, and the filer reads the file it first found on every reload.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-19 13:19:46 -08:00
f5e091cf91 wdclient, s3: return an embedded S3 gateway to its local filer after failover (#12772)
* wdclient: sort newly discovered filers and log every current-filer switch

Discovery appended new filers in map iteration order, so after the current
filer left, the fallback at index 0 was a random filer. Append them sorted.
Route every change of the current filer through one helper that logs it at
V(0) with its reason (failover, return_home, discovery), so an operator can
see which filer a gateway uses without new metrics.

* wdclient: prefer a home filer and return to it after failover with backoff

A FilerClient with FilerClientOption.HomeFiler set goes back to that filer
once its circuit breaker allows a re-check, instead of staying on whichever
peer it failed over to. Leaving home again soon after a return doubles the
wait before the next return, up to 5 minutes; a return that holds for 10
minutes resets it. Volume lookups and discovery follow the same preference.

* s3: embedded gateway uses its local filer as home filer

weed filer -s3, weed server -s3 and weed mini now build the gateway's filer
address in the form the filer registers in the master (ip:port.grpcPort),
with the gRPC port default applied, and pass it to the gateway as its home
filer, so the gateway returns to its local filer after any failover.
Standalone weed s3 has no home filer and behaves as before.

Refs #12768, follow-up to #12771

* wdclient: switch the current filer only from the filer a request started on

A volume lookup that started on a peer and finished after the client had
returned home moved it back to the peer and counted that as leaving home,
doubling the return backoff although home never failed. Concurrent
failovers could also each count as a separate departure.

Failovers now name the filer they started on and take effect only while it
is still current, via compare-and-swap, so only the winning switch logs and
feeds the backoff. S3's failover walk uses the same rule.

* wdclient: switch the current filer and record the return backoff in one step

The index moved by compare-and-swap before the switch was recorded, and
both ran under the shared read lock. A request reading the current filer
in that gap saw the peer with the old eligibility time and, with home's
breaker still closed after one failure, switched straight back home.
The eligibility check, the move and the bookkeeping now all run under
homeMu.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* wdclient: discovery falls back past a home that is still backing off

When discovery removed the current filer and home was not yet eligible,
the client took index 0, which is home whenever it is listed first. With
home's breaker still closed after a single failure, requests went back to
the filer they had just left. Take the next filer instead.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* wdclient: start return timing on every switch back to the home filer

Only a return_home switch stamped lastReturnHome. Discovery or a peer's
failover could also put the client back on home, and a failure soon
after such a return kept the backoff at its base value instead of
doubling it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Konstantin Lebedev <whitefox@mayflower.work>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-19 13:18:57 -08:00
Chris LuandClaude Opus 5.5 c9f3a02b94 filer: recover a leveldb store whose table a power loss damaged (#12780)
* filer: recover leveldb without rebuilding broken tables

leveldb.Recover rebuilds a table with damaged blocks using the user
comparer and filter on internal keys. The shortened index keys it writes
are not valid internal keys, so a Get on the rebuilt table panics, and
the bloom filter it writes misses every lookup. Recover with a comparer
that keeps full index keys (same name, so the manifest still matches)
and no filter.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* filer: recover a leveldb store when a compaction finds a corrupted table

When a compaction reads a damaged table block, as a power loss can
leave one, goleveldb latches the corruption error and returns it for
every later write until the database is reopened. A plain reopen loads
the same table and the next compaction fails again, so the store kept
rejecting inserts even across restarts.

Treat a corruption error from a write like a storage error, and reopen
through leveldb.Recover, which rebuilds the table without the damaged
blocks, then retry the write.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* filer: rebuild damaged leveldb tables before Recover

leveldb.Recover's own table rebuild orders internal keys with the user
comparer, so a damaged table holding several versions of one key fails
with "keys are not in increasing order" and the recovery gives up.
Overriding the comparer cannot fix that, because the same comparer
also serves the recovered database's user keys.

Rebuild every table that has corrupted blocks first, using an internal
key comparer and full index keys, then let Recover rebuild the manifest
from tables it no longer needs to rewrite. Seed the test tables from a
journal flush so they keep several versions of a key, and check keys
the write test does not rewrite.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* filer: leave unreadable leveldb tables for Recover to drop

table.NewReader reports a truncated table or a damaged footer through
the reader, not its error, so rebuildDamagedTables never stopped on one,
but it rewrote it as an empty table first. Skip tables with nothing
readable and let Recover drop them, and test an unreadable table next
to a healthy one.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* filer: let StrictRecovery drop damaged leveldb tables

With StrictRecovery set, leveldb.Recover drops a table that has corrupted
blocks. Salvaging the readable blocks first would hide the damage from it,
so skip the rebuild under that option.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* filer: hold the leveldb storage lock while rebuilding tables

The table rebuild ran after the failed open released the storage lock
and before Recover took it, so it could rewrite the tables of a database
another caller had open on the same storage. Take the lock for the
rebuild; Recover takes it again afterwards.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-19 13:13:30 -08:00
6bdcd17ed4 master: reject an assign on a follower as Unavailable (#12779)
* master: reject an assign on a follower as Unavailable

Assign on a master that is not the raft leader returned
raft.NotLeaderError as a plain error, which reaches gRPC clients as
codes.Unknown. operation.Assign retries only Unavailable,
ResourceExhausted, Canceled and DeadlineExceeded, so every assign made
during a leader election failed at once instead of waiting for the new
leader.

Return it as codes.Unavailable, like the warming-up status in the same
file, so clients retry within their 30 s assign budget. The message is
unchanged, so callers that match "Not current leader" still work.
StreamAssign now answers such a request in-band with the error message
instead of ending the stream, as it already does for warm-up.

TestAssignOnFollowerIsUnavailable covers it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* pb: keep the cached channel when the peer answers Unavailable with details

gRPC's own Unavailable (dial failure, closing transport) never carries
details, so one that does came from the peer over a working channel.
Closing the shared ClientConn for it cancels every other in-flight RPC
on it with "the client connection is closing".

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* master: tag the follower's assign rejection with a NOT_LEADER ErrorInfo

The detail marks it as the master's answer, so clients retry it
without tearing down their cached channel to that master.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
2026-11-19 13:11:40 -08:00
Chris LuandClaude Opus 5.5 729b872f01 test: let the merge-read FUSE test wait out the master's vacuum drain (#12777)
* test: let the merge-read vacuum outlast the master's write drain

Before compacting a volume the master waits up to 30 s for writes it
assigned to that volume to drain. The test cut each /vol/vacuum call at
30 s and gave the vacuum 60 s, so one or two slow drains failed it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test: retry the merge-read vacuum until the master runs one

A 409 means another vacuum holds the master's lock; check the chunks
only after a vacuum this test asked for has finished.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-19 12:44:48 -08:00
Chris LuandClaude Opus 5.5 972f85159e util/http: probe dead replicas in the background and key no-answer dials by machine (#12776)
* util/http: probe an unanswering server in the background, not on a read

Once the retry interval passed, the first read put the dead server first
and waited out the full dial timeout before trying a live replica. Keep it
last and dial it from a background probe with a short timeout instead.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* util/http: try every server of a machine that does not answer last

Unreachability was kept per host:port, so a powered-off node running one
volume server per disk cost one stalled read per server before each was
known dead. A dial that times out or finds no route now marks the whole
machine; a refused connection still marks only that port and shows the
machine is up.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* util/http: probe an unanswering server with an HTTP request

A bare TCP connect cleared servers that accept connections but never send
response headers, and could not dial an address without an explicit port.
Probe with a HEAD request through the global client instead, and have the
tests wait for the probe to settle.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* util/http: keep a timed-out port last after a sibling port answers

A dial timeout stored only the machine verdict, so a success on another
port of that machine cleared it and the port that timed out went back to
the front. Record the port as well as the machine.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* util/http: classify refused and unroutable dials with Winsock errnos on Windows

Windows reports these as WSAECONNREFUSED, WSAEHOSTUNREACH and friends,
which errors.Is does not match against the syscall constants. Test the
refusal with a real dial to a closed port.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* util/http: wait for background probes through a WaitGroup in tests

A refused probe can finish before the test reads the claimed record, so
inferring completion from the record's timestamp could wait out the
deadline on a probe that already settled.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* util/http: key machine verdicts apart from hosts without a port

A host without an explicit port is also its machine name, so a sibling
port answering cleared the timed-out host along with the machine verdict.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* util/http: group a bracketed port-less IPv6 host with its machine

net.SplitHostPort rejects "[2001:db8::9]", so the machine key kept the
brackets and missed the verdict stored for "[2001:db8::9]:9501".

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-19 12:44:10 -08:00
ed7324611c filer: metrics for metadata subscription delivery (lag, holds, ring fall-offs, claim and watermark ages) (#12773)
* filer: metrics for metadata subscription delivery (lag, holds, ring fall-offs, claim and watermark ages)

* filer: measure subscribe delivery lag after the real send using the mutation stamp, and record a watermark hold once per hold

* filer: skip empty progress markers when sampling subscribe delivery lag

* filer: assert subscribe delivery lag is sampled only after the stream send succeeds

* grafana: panels for the metadata subscription delivery metrics

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* filer: record a watermark hold when the read passes the held entry, not when the next hold starts

* filer: use a monotonic clock for the subscribe hold duration

---------

Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-19 12:43:34 -08:00
Chris LuandClaude Opus 5.5 05a1463855 wdclient: keep the embedded S3 gateway on its local filer across discovery (#12771)
* filer: keep an explicit gRPC port in the embedded gateways' filer address

runFiler handed the embedded S3/WebDAV/IAM/SFTP gateways
NewServerAddress(...).String(), and String() drops the .grpcPort part.
With a non-default -port.grpc the gateways then dialed port+10000 and
did not recognize their own filer in the master's list.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* wdclient: match discovered filers by gRPC address

A filer listed as host:port and the same filer discovered as
host:port.grpcPort were compared as raw strings, so a refresh pruned the
configured filer, added every discovered filer as new in map order, and
reset the current filer to whichever came first; a pushed join added it
a second time. Key filers by their gRPC address: a survivor keeps its
health and current position and takes the discovered address form, and
pushed joins and leaves match the same way.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-19 12:42:31 -08:00
c1c95e127f filer: scoped read-only tokens for metadata subscriptions (#12767)
* security: scoped read-only token for filer metadata subscriptions

A token signed with jwt.filer_signing.subscribe.key names the subscribe
methods and the path prefixes its holder may see. mTLS peers and admin
tokens stay unrestricted. Internal subscribers attach an admin token
when both keys are configured.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* filer: enforce subscription token scope on metadata streams

With jwt.filer_signing.subscribe.key set, SubscribeMetadata and
SubscribeLocalMetadata authorize the caller first. A scoped caller gets
a stream that sends only events whose paths are all under its prefixes,
filters batches, and never carries log file refs.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* filer: send an admin token on internal metadata subscriptions

Peer filers, metadata followers and the lifecycle reader attach an
admin token when the subscribe key is configured, so they keep the full
stream without mTLS.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* shell: add fs.meta.token to mint metadata subscription tokens

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* filer: end scoped metadata subscriptions at token expiry

The scope now carries the token's exp. The scoped stream's context ends
at that time and the call returns Unauthenticated, so an open stream no
longer outlives its token.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* filer.sync: sign metadata subscriptions with the source cluster's key

MetadataFollowOption takes a FilerJwtProvider; filer.sync passes the
one loaded from -a.security or -b.security, so clusters with different
admin keys both accept the subscription.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* shell: report fs.meta.token flag errors

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* filer: give scoped metadata subscribers their own client id

A scoped caller could reuse another subscriber's ClientId with a higher
epoch and replace that registration, disconnecting it. Scoped streams now
register under a server-chosen id that no other subscriber holds.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* security: accept only HS256 metadata subscription tokens

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* security: test HS384 metadata subscription token rejection

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* filer: never let a scoped subscriber replace another registration

The server-chosen id can be taken by another subscriber between choosing
and registering it. Registering scoped streams with the lowest epoch makes
addClient refuse them rather than replace the other registration.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* filer: send position markers for events a scoped token withholds

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* filer: test a scoped stream withholding a whole event batch

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* filer: filter SendMsg on scoped metadata streams

The scoped wrappers only overrode Send, so a SendMsg on the embedded
stream would reach the client unfiltered. Route it through the same
filter and refuse any other message type. Ported from #12790.

Co-authored-by: mindaeon <30555587+mindaeon@users.noreply.github.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* filer: test a scoped subscription through the real subscribe loop

Run SubscribeLocalMetadata with a scoped token over a flushed backlog,
the in-memory tail and live events: only in-scope events arrive, sibling,
"..", encoded and cross-boundary rename paths are dropped, and no log
file references are sent although the client asks for them. Without the
subscribe key every event still arrives. Ported from #12790.

Co-authored-by: mindaeon <30555587+mindaeon@users.noreply.github.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* filer: test that the HTTP API refuses metadata subscription tokens

GET, HEAD, PUT, POST and DELETE all refuse a subscription token, also
when the subscribe key equals the read or write key. Ported from #12790.

Co-authored-by: mindaeon <30555587+mindaeon@users.noreply.github.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: mindaeon <30555587+mindaeon@users.noreply.github.com>
2026-11-19 12:41:16 -08:00
Chris LuandClaude Opus 5.5 668ca45029 s3: ListParts returns each part number once, its latest attempt (#12770)
* s3: list each part number once, its latest attempt

Every UploadPart writes its own NNNN_<uuid>.part entry, so a part number
uploaded twice listed twice, superseded ETag included, and a max-parts page
could hold only the stale attempt. ListParts now groups the entries by part
number, reports the attempt sortEntriesByLatestChunk ranks first (the same
order completion uses), and pages over distinct part numbers, reading past a
page boundary that splits one part number's attempts.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3: test ListParts over repeated part attempts

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3: rank an empty part attempt by commit stamp or mtime

An empty UploadPart body stores no chunks, so sortEntriesByLatestChunk gave
it time zero and an empty re-upload always lost to the attempt it replaced.
Attempts now rank by the filer's commit stamp when both carry one, else by
the later of the first chunk's write time and the entry mtime.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3: keep only the newest attempt per part while listing parts

listPartAttempts held every attempt of a part number until the page was
built, so a heavily retried part grew one ListParts request's memory past
what max-parts bounds. It now compares each entry against the newest seen
for its part number with compareLatestPartAttempt, the comparator
sortEntriesByLatestChunk sorts with.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3: order part attempts by commit stamp, then attempt time

Comparing commit stamps only when both attempts carried one made the order
non-transitive across stamped and unstamped attempts, so listing and
completion could pick different attempts depending on input order. The
commit stamp now leads (unstamped entries predate the field) and the
attempt time breaks ties.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3: list part 10000 after the four-digit part numbers

Part names sort as text, so 10000_<uuid>.part lands between 1000.part and
1000_<uuid>.part. A page could end on part 10000 and the next marker then
skipped parts 1000-9999, and a legacy 1000.part could be reported before
the newer attempt of part 1000 was read. List the four-digit part numbers
first and part 10000 in its own prefix listing.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3: test ListParts paging across part 10000

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-19 12:36:27 -08:00
Chris LuandClaude Opus 5.5 dcbea0626f test/s3: patch s3-fast-list's data map race in the listing study build (#12766)
test/s3: build s3-fast-list with its data map draining the last page

The data map task reads the channel and then the list-task state. When
the list task sends its last page and completes between the two reads,
the map is dumped without that page and the tool exits 0, so a one-page
prefix comes back empty. Read the state before the channel.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-19 12:21:32 -08:00
Chris LuandClaude Opus 5.5 de4075660f telemetry: refresh the server module's go.mod so the integration test builds (#12765)
* telemetry: refresh the server module to the root module's dependency versions

The server module replaces seaweedfs with ../.., so root dependency
bumps raised prometheus/common, prometheus/procfs and x/sys past the
versions its go.mod listed, and go build refused to run.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* ci: run the telemetry integration test on root go.mod changes

The telemetry server module builds against the root module through a
replace directive, so a root dependency bump can leave its go.mod stale
without touching telemetry/.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-19 00:55:09 -08:00
Chris LuandClaude Opus 5.5 652133a28a telemetry: keep the newest release in the versions chart (#12764)
The versions chart keeps the 8 versions with the most clusters on the last
day and folds the rest into "other". A new release starts small, so 4.54
(282 clusters) was folded in while 4.53 shrank as clusters upgraded,
which showed as a dip across the named versions. The newest plain release
now always keeps its own band.


Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-19 00:08:19 -08:00
e012fca9a2 mount: keep lookups from stalling behind directory cache rebuilds (#12754)
* mount: release the meta cache lock between batches of a rebuild wipe

deleteFolderChildrenForRebuild held mc's write lock while it listed every
cached child of the directory and deleted them. Every metadata read in the
mount (FindEntry, IsNameFresh, isOversized) takes the read lock, and a
waiting writer blocks new readers, so while a large directory was wiped,
lookups and getattrs in unrelated directories queued behind the scan. The
scan gets slower as leveldb accumulates the deletion markers of earlier
wipes of the same directory, which happens quickly once lookups rebuild
expired directories.

Wipe rebuildWipeBatchSize (1024) children per lock hold and drop the lock
between batches. A child is judged pinned under the same lock that deletes
it, so a local-only create is kept whether it was pinned before the wipe
started or between batches. The listing that refills the directory already
ran outside the lock, so batching adds no interleaving the build did not
allow.

Add LevelDBStore.BatchDeleteEntries, the counterpart of BatchInsertEntries.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>

* mount: bound how long a lookup waits on the directory rebuild it starts

Since #11268 a lookup whose parent's cache expired rebuilds the parent and
every expired ancestor up to the mount root before it answers, and every
lookup beneath them joins the same builds. With a slow listing (a large
directory, a busy filer, or the meta cache lock held by another rebuild) a
plain stat, an ls -l, or the parent touch after mkdir waited for all of it.

Wait at most lookupRebuildTimeout (2s), then fall back to one LookupEntry
RPC. Since #12206 the shared build does not run on the requester's context,
so it keeps going after the waiter gives up and later lookups find the
directory cached. A timed-out wait records a rebuild attempt, so lookups
during the cooldown do not queue on a build that is already known to be
slow.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>

* mount: wait for the rebuild's listing in TestLookupDoesNotWaitOutSlowRebuild

The build runs on its own goroutine, so the lookup that stopped waiting
for it can return before the build's ListEntries reaches the filer. The
test read `started` without waiting right after the lookup returned, which
relied on an ordering nothing guarantees, and the ListEntries count taken
afterwards assumed the same. Wait for the listing to arrive, bounded by
the test's deadline.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>

* mount: keep children inserted while a rebuild wipe runs

A deferred create inserts its placeholder before Create registers the
handle that pins it. The single-lock wipe made such an insert wait until
it finished, so the placeholder survived; a batch that runs in that gap
judged it unpinned and dropped it, and the rebuilt listing then omitted
the file until its flush.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* mount: drop a cached path that an event vacates during a rebuild wipe

A build replays only the events later than its listing's snapshot and
leaves the earlier ones to the listing, which starts after the wipe. With
the wipe in batches a file's whole life fits inside it: created, flushed,
then unlinked or renamed away. Both events are earlier than the snapshot
and are skipped, the listing has no such file, and the placeholder the
wipe kept stays in the published directory until the next rebuild.

While a directory's wipe runs, an event buffered for it now takes the
path it vacates out of the store at once, fenced by versions like any
other event. After the wipe the store holds what the listing put there,
and a removal earlier than the snapshot may still arrive, so from then on
the event is left to the replay as before.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>

* mount: check which file a removal took away during a rebuild wipe

Two things the removal applied during a rebuild wipe got wrong.

A placeholder is unversioned, so the version gate lets every removal
through. One that arrived late, for an earlier file of the same name,
took out the placeholder of a file created since. The entry the event
took away is now compared with the cached one, by inode or, with no
inode to compare, by creation time. An event that only names the file
cannot tell and is taken at its word.

The entry was deleted without OP=MV, so taking one name of a hard-linked
file out of the cache also took one off the count its other names share.
A replayed event that vacates a path already deletes with OP=MV.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>

* mount: keep a removal the rebuild wipe declined out of the replay

The wipe declines a removal when what is cached there is newer than the
event or is another file than the one it took away. The event stayed
buffered anyway. An unversioned listing clears the children's version
records and replays every buffered event, so the declined removal then
deleted the entry the wipe kept and the directory was published without
it.

A declined removal now buffers only the name it puts in place, so a
rename still lands on replay.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* mount: check a declined removal again when the build replays it

The previous commit dropped the removal half of an event the wipe
declined, judged by the file cached when the event arrived. Earlier
buffered events can replace that file during the replay: a listing
attempt that failed leaves its events buffered, so the retry's wipe may
see the first attempt's file while the replay puts a later one in place
first. The dropped delete then left that later file published.

The wipe now records which removals it declined, and the replay checks
each against the entry it finds at that point, replaying only the name
the event puts in place when that entry is still another file. A
removal declined by version is replayed as before.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* mount: keep a declined removal's invalidation off the placeholder's handle

After the replay the build invalidates every buffered event again,
raised to the snapshot, and it still sent the original of a removal the
replay rejected. A late delete of an earlier file then reached the
handle of the placeholder at that name: the placeholder is unversioned,
so the raised timestamp got past it and the handle was marked deleted.

The replay's decision on a declined removal now replaces the event in
the buffer, and the invalidations use what was replayed. It is made
before the snapshot check, so a removal the listing covered is judged
the same way.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* mount: keep an event's signatures on the halves a rename is split into

A declined removal replays as its rename's create half, and a rename across
a building directory is split in two. Both dropped the event's filer
signatures, so a handle stamped by that filer no longer fenced the replayed
invalidation by timestamp and could take older listed contents over newer
ones it had read.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* mount: drain invalidations around the declined-removal build check

Invalidations run on a background worker, so the baseline could miss the
removal's own and the final read could run before the build's arrived.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: MapleWhiteZZZ <18653045+MapleWhiteZZZ@users.noreply.github.com>
Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
2026-11-19 00:04:03 -08:00
Chris LuandClaude Opus 5.5 3159b8408c filer, volume: fsync chunk deletes under an fsync path rule (#12759)
* volume: fsync a batch delete that asks for it

BatchDeleteRequest gets an fsync field, and the Go and Rust volume
servers pass it to the needle delete, which already orders the
tombstone, .dat sync, publish and .idx sync for a durable delete.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* operation: pass an fsync flag to batch deletes

DeleteFileIdsWithLookupVolumeId and DeleteFileIdsAtOneVolumeServer take
an fsync argument and send it in the BatchDeleteRequest. Every caller
passes false for now.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* filer: fsync chunk deletes under an fsync path rule

DeleteChunks passes the path rule's fsync flag to the deletion queue.
The flag is kept with the pending record until the delete is
confirmed, so retries and re-queues keep it, and the batch and retry
loops send marked ids in a durable BatchDelete.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* filer: keep fsync marks in the deletion ledger

Marked ids go into the trailing parts of a manifest ledger, counted by
fsyncParts. Older filers read those parts as plain ids and still delete
them; a reload or a claimed foreign ledger restores the marks. A ledger
without marks keeps the single array form.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* filer: record a queued deletion before the delete loop can see it

queueDeletions published the ids to the hot queue before taking the
ledger lock, so a concurrent flush could delete an id before its fsync
mark landed and then forget the mark with the record.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* filer: fsync a folder delete's chunks by each child's path rule

A recursive delete collected its file children's chunks and resolved
the rule once for the folder, which misses a trailing-slash rule on
the folder itself and file-prefix rules. Each file child's own rule now
decides fsync; the folder rule still decides chunk deletion.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* filer: keep a newer fsync request when a plain delete finishes

An id re-queued with fsync while a plain delete of it was in flight
lost its mark when that delete succeeded, so the queued copy also ran
without fsync. A plain attempt now leaves a marked record for its
queued copy; a durable delete of an already-deleted needle still syncs
the tombstone.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* filer: delete a folder's plain child chunks without fsync

Children whose own rule turns fsync off were still deleted durably when
the folder's rule asked for fsync. Only the children marked by their
own rule now go out with fsync.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* filer: retry a durable chunk delete the volume server failed

A volume server rolls back a durable delete it cannot sync and answers
with a 5xx carrying the I/O error text, which the classifier took as
permanent, so the record and its fsync mark were dropped while the
needle stayed live. A 5xx answer to a durable delete is now retryable,
within the usual attempt limit.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-19 00:02:43 -08:00
Chris LuandClaude Opus 5.5 f88751c3ff s3api: evaluate request condition keys for STS sessions (#12762)
* s3api: build request condition keys in one helper

Split the keys ExtractConditionValuesFromRequest reads from the request
itself into RequestConditionValues, keeping aws:SourceIp and the
bucket-listing keys in the method.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3api: evaluate request condition keys for STS sessions

The IAM integration context for STS and OIDC sessions carried only the
source IP, legacy user agent/referer keys and claims, so role and session
policy conditions on s3:x-amz-*, aws:SecureTransport, aws:CurrentTime,
s3:RequestObjectTag/* and the other request keys never matched: a Deny
on them never fired, and a negated Deny always did. Add the same request
keys the static identity path evaluates.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3api: report SigV4 presigned requests as REST-QUERY-STRING

s3:authType recognized only SigV2 query authentication, so a policy
condition on REST-QUERY-STRING never matched a SigV4 presigned URL.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-19 00:01:32 -08:00
Chris LuandClaude Opus 5.5 2089abecf3 ci: cache Snowflake suite Maven deps and fall back to a Central mirror (#12763)
* ci: cache the Snowflake suite's Maven dependencies under their own key

setup-java's maven cache key hashes the repository pom.xml files, which
other Java workflows share. The Snowflake job restored that cache, which
lacks the suite's plugins and dependencies, and never saved its own because
the key already existed. Every run downloaded the whole suite dependency
tree from Maven Central and failed when Central answered 429.

Key the cache on run.sh, which pins the suite revision.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* ci: fall back to Google's Maven Central mirror in the Snowflake job

A cold cache still resolves about 36 MB of plugins and dependencies, and
Maven gives up after three 429 retries within 30 s. Add the mirror as a
settings-profile repository; Maven tries it first and moves on to Central
for any artifact the mirror fails to serve.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-19 00:00:41 -08:00
Carlos Leyva 6d0a6e0935 filer: mint read_auth for the data chunks behind manifest chunks (#12756)
* filer: mint read_auth for the data chunks behind manifest chunks

* filer: cover every chunk behind manifests in read_auth (data + nested), minted after resolution, and skip when read signing is off

* filer: pass the master client as manifest cache invalidator, and move the read_auth helpers below populateBatchReadAuth for the enterprise merge
2026-11-18 19:37:06 -08:00
Chris LuandClaude Opus 5.5 9b94f00325 test: mount pinned Iceberg jars into the Spark catalog tests (#12757)
* testutil: fetch pinned Maven Central jars with a mirror fallback

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test: mount pinned Iceberg jars into the Spark catalog containers

Each test started a new Spark container that resolved the Iceberg runtime
and AWS bundle through Ivy from Maven Central, so a run downloaded them six
times and a single refused request left Spark retrying until the package
timed out.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* ci: cache the Iceberg jars for the Spark catalog tests

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* testutil: test the verified jar download

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test: bound Maven jar downloads by one deadline shared across retries

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test: give each Maven mirror attempt its own timeout within the download budget

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-18 19:35:54 -08:00
Chris LuandClaude Opus 5.5 d2a0002869 s3api: fix listing regressions from argument validation (empty continuation-token, V1 Prefix encoding) (#12760)
* s3api: accept an empty ListObjectsV2 continuation-token

An empty continuation-token lists from the start and is echoed back
empty, as S3 does; rejecting it broke s3-tests
test_bucket_listv2_continuationtoken_empty.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3api: leave the ListObjects V1 Prefix unencoded under encoding-type=url

S3 encodes the V1 Delimiter, markers and keys but echoes Prefix as sent,
and botocore, which always asks for url encoding, decodes the V1 Prefix
only for V2. Encoding it broke s3-tests test_bucket_list_prefix_unreadable.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3api: use a codespell-clean key in the versions encoding test

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3api: URL-encode a V1 Prefix that XML cannot carry

Leaving the V1 Prefix raw under encoding-type=url turned a control
character such as \x01 into U+FFFD in the response. Such a prefix is
URL-encoded again; prefixes that are valid XML text, including '\n',
stay as sent.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-18 19:34:57 -08:00
github-actions[bot] c01e87689e docs: regenerate star history chart 2026-11-19 02:58:55 +00:00
Chris LuandClaude Opus 5.5 7b4bdb77bb master: export writable volumes per disk and reset leader-only layout gauges on followers (#12751)
* master: export writable volumes per disk for each volume layout

The layout gauges only report how many volumes of a collection take
writes, not where they sit, so two writable volumes sharing one disk
cannot be seen or alerted on. The leader now exports
writable_volumes_per_disk with the layout labels plus data center, rack,
node and disk id, counting the replicas of the layout's writable volumes
on each disk (state=writable) and those of them that are crowded
(state=crowded). Series for disks that no longer hold a writable volume of
the layout, and for layouts that are gone, are deleted.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* master: reset the leader-only layout gauges on followers

The layout writable, crowded, under-replicated and per-disk gauges are only
set by the leader, so a master that lost leadership kept exporting the
values of its last term indefinitely, and dashboards that aggregate over
masters could show numbers days old. A master that is not the leader now
clears them on each periodic pass.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* grafana: chart disks holding several writable volumes of a collection

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* grafana: keep disks of different clusters apart in the per-disk panel

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-18 16:17:04 -08:00
Chris LuandClaude Opus 5.5 1fd5008d63 s3api: listing argument validation and encoding-type=url for versions, prefix and delimiter (#12753)
* s3api: reject an empty ListObjectsV2 continuation-token

AWS answers continuation-token= with 400 InvalidArgument ("The continuation
token provided is incorrect"); SeaweedFS listed from the start of the bucket.
No token this server issues is empty, and raw-key and base64: tokens keep
resuming as before.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* s3api: reject an encoding-type other than url on listings

AWS answers any encoding-type other than url with 400 InvalidArgument
("Invalid Encoding Method specified in Request") on ListObjects,
ListObjectsV2, ListObjectVersions, ListMultipartUploads and ListParts;
SeaweedFS ignored the value and listed unencoded keys.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* s3api: apply encoding-type=url to ListObjectVersions

The handler accepted encoding-type=url but returned raw keys. It now
URL-encodes Key in Version and DeleteMarker entries, CommonPrefixes, Prefix,
Delimiter, KeyMarker and NextKeyMarker with the urlPathEscape form ListObjects
uses, and echoes EncodingType.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* s3api: URL-encode Prefix and Delimiter in ListObjects responses

Under encoding-type=url, ListObjects and ListObjectsV2 encoded keys,
CommonPrefixes, markers and StartAfter but echoed Prefix and Delimiter raw,
so a client decoding them once corrupted a prefix holding '%' or '+'.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-18 15:55:41 -08:00
646549eee1 volume growth: spread writable volumes of a layout over servers and disks (#12749)
* volume growth: spread writable volumes of a layout over servers and disks

Volume growth picked every server, and the volume server picked every disk,
independently: the server weighted by free slots, the disk by the lowest
total volume count. A grow request therefore stacks several writable volumes
of one collection on some disks while most disks get none (a Poisson spread:
with 160 replicas on 479 disks about 20 disks get two or three), and their
writes share those disks.

Data centers and racks are still picked weighted by free slots. Inside a
rack the server is now one of those holding the fewest writable volumes of
the layout per disk, weighted by free slots among them; this covers the main
server, same-rack replicas and replicas walked into other racks and data
centers. On the volume server the disk is the one with the fewest writable
volumes of the collection, then the lowest volume count as before.

A grow request registers each volume before placing the next, so a request
smaller than the number of disks lands on distinct disks.

Fixes #12746

* volume growth: scan the layout load once per grow request

Placing each volume rescanned all writable volumes of the layout under its
read lock. RecordAssign takes that lock exclusively on every assign, so with
thousands of writables (0.8 ms for 10000) every volume of a grow request
stalled assigns once more.

GrowByCountAndType now scans once and findAndGrow adds every volume it
creates to the counts. Per volume, placement costs about 130-160 us on 480
servers whatever the number of writables, against 120 us before the spread
change; the single scan is 0.17 ms for 2000 writables and 0.8 ms for 10000.

* volume growth: address review of the spread placement

- Count only disks that report volume slots in a server's load. An
  unavailable disk, or one low on space with no volumes, is still listed in
  DiskTags with no slots; counting it understated the load of a server with
  failed disks and piled volumes onto its remaining ones.
- On the volume server, judge a volume full by its .dat size, which the
  master compares with the limit, not by ContentSize.
- A server that refuses an allocation ranks last for the rest of the grow
  request, and the request retries the volume up to 3 times. The least
  loaded server is picked first deterministically, so without this one server
  that cannot fit a preallocated volume stopped the layout from growing.
- Scan the layout load after warmup, so a grow request issued right after a
  leader change sees the writable volumes the returning servers report.

* volume growth: keep the request-local placement load ahead of the request fields

The JSON request fields stay together at the end of VolumeGrowOption, and
the comment now says the load lives for the whole grow request.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* volume(rs): place a new volume on the disk with the fewest writable volumes of its collection

Store::add_volume picked the disk with the most free slots, so the Rust
volume server ignored the collection spread rule the Go server now applies.
find_free_location ranks disks by the writable volumes of the collection
first (not read-only, .dat below the volume size limit), then by free slots.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* volume growth: leave a refusing server out of every placement tier

A server that refused an allocation only ranked last among the servers of
its rack, so the weighted data center and rack picks could still choose its
rack on every retry while other racks had room. It now offers no free slots
for the rest of the grow request, which takes its slots out of the rack and
data center weights too. If only refusing servers are left, they are tried
again.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* volume growth: require the full same-rack count only in the main rack

The data center check asked every rack for SameRackCount+1 free servers,
but only the main rack holds the same-rack copies; each other rack takes one.
With a refused server left out of its rack, the check rejected a feasible
placement, the retry put the refused server back, and it was picked again.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Konstantin Lebedev <whitefox@mayflower.work>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-18 15:51:13 -08:00
Chris LuandClaude Opus 5.5 9cd183b04f volume: fsync a needle delete that asks for it (#12736)
* volume: fsync a needle delete that asks for it

A delete now takes fsync like a write: the tombstone is synced before the
needle map drops the needle, and the .idx row before the delete is
acknowledged. A failed .dat sync takes the tombstone back off the .dat.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* volume: honor fsync=true on DELETE and forward it to replicas

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* volume(rs): fsync a needle delete that asks for it

Mirrors the Go volume server: DELETE with fsync=true flushes the tombstone
before the index drops the needle and the .idx before the ack, and a failed
.dat flush takes the tombstone back off the .dat.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* volume: let a durable delete through on a full volume

A queued delete hit the batch worker's volume size check, which a
tombstone append is exempt from inline, so fsync=true deletes failed once
the volume reached its hard limit.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* volume: sync a durable delete of an already-deleted needle

An earlier delete without fsync may have dropped the needle with its
tombstone still unsynced, so the inline path syncs before acking anyway,
as the batch worker already does.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* volume: refuse a batch request that races the worker stopping

stopWorker can close the batch channel between startWorker handing it out
and the send, which panicked. The request now falls back to the inline path.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* volume(rs): sync a durable delete of an already-deleted needle

Mirrors the Go volume server: an earlier delete without fsync may have
dropped the needle with its tombstone still unsynced.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* volume(rs): run a durable delete on a blocking thread

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* volume: keep syncDelete's signature for existing callers

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* volume: refuse a write a delete cancelled before it appends

A write cancelled while still queued behind the volume locks used to
append, publish and then tombstone itself without a sync, after its
delete may already have been acknowledged as durable. A crash could
keep the row and lose the tombstone. Refuse it before the append.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* volume(rs): refuse a write a delete cancelled before it appends

Mirrors the Go change: a write cancelled while still queued behind the
volume locks no longer appends, publishes and tombstones itself without
a sync after its delete may already be durable.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* volume(rs): drop the write guards for a durable delete's flushes

The durable delete held the write lease and the io write guard across
both fsyncs, so reads of the volume waited on them. Append and publish
the tombstone under the guards and flush with only the io read side, as
write_synced_run and the Go path already do. do_delete_request keeps its
signature.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* volume: stop taking writes when a durable delete fails to index

The needle map drops the needle before it appends the .idx row, so after
a failed append a durable retry found nothing to delete and acked, and a
later write could bury the unindexed tombstone so a reload brought the
needle back. Fence the volume the way a failed durable write does.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* volume(rs): stop taking writes when a durable delete fails to index

Rust counterpart of the Go change: the needle map drops the needle
before it appends the .idx row, so a failed append left a durable retry
nothing to delete, and a later write could bury the unindexed tombstone.
publish_delete_request now fences the volume for a durable delete, as
publish_write does.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* volume: stop taking writes when any delete fails to index

A non-durable delete whose .idx append failed left the volume writable with
the needle already dropped from the map, so a later durable retry found
nothing to delete and acknowledged without the missing row.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* volume(rs): stop taking writes when any delete fails to index

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* volume: take the fsync flag on the existing delete functions

DeleteVolumeNeedle, deleteNeedle2 and syncDelete take fsync instead of
gaining *WithFsync variants.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* volume(rs): take the fsync flag on the existing delete functions

delete_volume_needle and delete_needle take fsync instead of gaining
*_with_fsync variants; the durable path and the deletable checks fold
back into delete_needle, and the cancelled-write check mirrors Go's
isCancelledWrite.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-18 15:18:29 -08:00
Chris LuandClaude Opus 5.5 23eef59417 test/s3: run the s3-listing-study listing tools and strategies in CI (#12750)
* test(s3): s3-listing-study integration suite (WIP)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* test(s3): run the s3-listing-study tools against SeaweedFS (WIP)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* test(s3): s3-listing-study deleted-prefix probe, fresh data dir per run

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* test(s3): listing study ports keep clear of every gRPC companion

free_ports.py compared only base ports, so a base port could land on
another server's base+10000 gRPC listener.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test(s3): listing study ports stay below the ephemeral range

Base ports of 20000-55000 put both them and their gRPC companions inside
32768-60999, so an outgoing connection made while weed server starts could
take a port before the server that was given it binds.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test(s3): listing study checks the order inside each listing page

parse_page sorted Contents and CommonPrefixes together before the probes
compared them, so keys returned out of order within one page passed. Check
each list's byte order first; the versions pager does the same.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test(s3): listing study applies --only before running a probe

The filter was applied to the probe results, so a run limited to one tool
still sent every reference listing request and the deleted-prefix check.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test(s3): listing study requires CommonPrefixes from shallow modes

The CommonPrefixes comparison ran only when a tool returned at least one
prefix, so a delimiter listing of the canary root that came back empty
passed. Compare them for every shallow mode except the two whose
normalizer keeps keys only.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-18 15:17:25 -08:00
Chris LuandClaude Opus 5.5 04c7b12238 s3api: reject unknown notification elements; fail test-with-server on a stuck shutdown (#12752)
* s3api: reject unknown elements in PutBucketNotificationConfiguration

xml.Unmarshal skips children that match no field, so a configuration
whose only child was an unmapped element parsed as empty and returned
success while nothing was stored.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test/s3/tigris_compat: fail test-with-server when the server does not stop

A passing suite returned success even when stop-server gave up with the
server still running.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-18 15:15:38 -08:00
Chris LuandClaude Opus 5.5 f5418845ce test/s3: run tigrisdata-community s3-api-compat-tests in CI (#12741)
* test/s3: run tigrisdata-community s3-api-compat-tests in CI

Runs the suite, pinned to a commit, against weed mini through a small
provider patch, and fails on any test that is not in
expected_failures.txt or on a listed test that starts passing.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* test/s3/tigris_compat: prepare the suite again when the patch changes

The checkout was reused whenever it sat at the pinned commit, so an
edited seaweedfs-provider.patch, or a checkout whose patch failed to
apply, kept being built as it was. Record the commit and the patch's
hash once the patch applies, and prepare the checkout again when either
differs.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test/s3/tigris_compat: keep the pid file of a server already running

start-server removed weed-server.pid before it found the port taken, so
a second start lost the pid of the server holding it and stop-server
could no longer stop that one. Remove the file only once the port is
free, and have test-with-server stop only a server it started.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test/s3/tigris_compat: bound the S3 port probes

A listener that accepts but never answers held curl, and with it the
readiness loop and the cleanup after it, for good. Give each probe two
seconds.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test/s3/tigris_compat: expect a listed test to end with its listed status

A listed test passed the check with any non-pass status, so a setup error
that skipped the call under test (RestoreObject's PutObject, say) went
unnoticed. Entries now expect fail unless they name another status; only
GetBucketAccelerateConfiguration, whose setup PUT is unsupported, expects
error.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test/s3/tigris_compat: refuse to start while the recorded server still runs

A second start-server before the first server answered on the S3 port
passed the port probe and dropped the pid file of the live server, so
stop-server could no longer stop it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test/s3/tigris_compat: recognize the test server by its command line

start-server and stop-server took any process named weed at the recorded
pid for the test server, so a weed process that reused the pid would be
reported as running and killed by stop-server. Match the command line
start-server launches instead.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test/s3/tigris_compat: recognize the test server by its start time

A weed mini started from another checkout has the same command line, so
a pid it reused still matched. Record the process start time next to the
pid at launch and require both before treating the process as the test
server.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test/s3/tigris_compat: PutObjectTagging_VersionId passes now

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test/s3/tigris_compat: keep the server records until the server exits

stop-server waited a fixed two seconds after SIGTERM and then removed
weed-server.pid and weed-server.started even if weed was still shutting
down, so a later stop-server could not find it and clean could delete
data it still used. Wait up to 30 seconds for the recorded server to exit
and keep the records, failing, while it still runs.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test/s3/tigris_compat: start the server on an empty data directory

A data directory left by an earlier run on other ports made the master
freeze volume growth (topology id mismatch), and every PutObject failed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-18 15:14:52 -08:00
Chris LuandClaude Opus 5.5 53bc62c773 s3api: accept the accelerate and notification configurations the bucket already reports (#12747)
* test/s3: run tigrisdata-community s3-api-compat-tests in CI

Runs the suite, pinned to a commit, against weed mini through a small
provider patch, and fails on any test that is not in
expected_failures.txt or on a listed test that starts passing.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* test/s3/tigris_compat: prepare the suite again when the patch changes

The checkout was reused whenever it sat at the pinned commit, so an
edited seaweedfs-provider.patch, or a checkout whose patch failed to
apply, kept being built as it was. Record the commit and the patch's
hash once the patch applies, and prepare the checkout again when either
differs.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test/s3/tigris_compat: keep the pid file of a server already running

start-server removed weed-server.pid before it found the port taken, so
a second start lost the pid of the server holding it and stop-server
could no longer stop that one. Remove the file only once the port is
free, and have test-with-server stop only a server it started.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test/s3/tigris_compat: bound the S3 port probes

A listener that accepts but never answers held curl, and with it the
readiness loop and the cleanup after it, for good. Give each probe two
seconds.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3api: accept PutBucketAccelerateConfiguration with Status=Suspended

GetBucketAccelerateConfiguration always reports Suspended, but the PUT
answered 501 for every request. Suspended is now accepted, Enabled
stays NotImplemented, and the accelerate subresource resolves to
s3:GetAccelerateConfiguration / s3:PutAccelerateConfiguration.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* s3api: accept an empty PutBucketNotificationConfiguration

GetBucketNotificationConfiguration always reports an empty
configuration, but the PUT answered 501, including for the empty
configuration clients send to clear notifications. An empty one is now
accepted; a topic, queue, Lambda or EventBridge destination stays
NotImplemented.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* s3api: reject accelerate and notification bodies with the wrong root

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* test/s3/tigris_compat: expect a listed test to end with its listed status

A listed test passed the check with any non-pass status, so a setup error
that skipped the call under test (RestoreObject's PutObject, say) went
unnoticed. Entries now expect fail unless they name another status; only
GetBucketAccelerateConfiguration, whose setup PUT is unsupported, expects
error.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test/s3/tigris_compat: refuse to start while the recorded server still runs

A second start-server before the first server answered on the S3 port
passed the port probe and dropped the pid file of the live server, so
stop-server could no longer stop it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test/s3/tigris_compat: recognize the test server by its command line

start-server and stop-server took any process named weed at the recorded
pid for the test server, so a weed process that reused the pid would be
reported as running and killed by stop-server. Match the command line
start-server launches instead.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test/s3/tigris_compat: recognize the test server by its start time

A weed mini started from another checkout has the same command line, so
a pid it reused still matched. Record the process start time next to the
pid at launch and require both before treating the process as the test
server.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test/s3/tigris_compat: PutObjectTagging_VersionId passes now

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test/s3/tigris_compat: keep the server records until the server exits

stop-server waited a fixed two seconds after SIGTERM and then removed
weed-server.pid and weed-server.started even if weed was still shutting
down, so a later stop-server could not find it and clean could delete
data it still used. Wait up to 30 seconds for the recorded server to exit
and keep the records, failing, while it still runs.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test/s3/tigris_compat: start the server on an empty data directory

A data directory left by an earlier run on other ports made the master
freeze volume growth (topology id mismatch), and every PutObject failed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-18 14:55:18 -08:00
Chris LuandClaude Opus 5.5 1f5cfc7552 s3api: return x-amz-version-id from object tagging on versioned buckets (#12745)
* s3api: return x-amz-version-id from object tagging on versioned buckets

PutObjectTagging, GetObjectTagging and DeleteObjectTagging now report
the version they acted on, "null" for an object written before
versioning was enabled.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* s3api: set the tagging x-amz-version-id only on success

The header was set before the tag-conditioned policy recheck, so a
denied request carried the version id on its AccessDenied response.
Also test the "null" version and require distinct version ids.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-18 14:30:55 -08:00
Chris LuandClaude Opus 5.5 b1c315b837 s3api: report no ownership controls after DeleteBucketOwnershipControls (#12748)
* test/s3: run tigrisdata-community s3-api-compat-tests in CI

Runs the suite, pinned to a commit, against weed mini through a small
provider patch, and fails on any test that is not in
expected_failures.txt or on a listed test that starts passing.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* s3api: report no ownership controls after DeleteBucketOwnershipControls

GetBucketOwnershipControls reports BucketOwnerEnforced for a bucket that
never stored a setting, so it kept reporting it after the setting was
deleted. The delete now records that it happened, and GET answers
OwnershipControlsNotFoundError until ownership is put again. How
objects are written after the delete is unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* s3api: answer DeleteBucketOwnershipControls with 204 No Content

AWS answers a successful DeleteBucketOwnershipControls with 204 and no
body; this answered 200 with an empty OwnershipControls document.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-18 14:25:15 -08:00
aceb089bc8 lance: per-table cleanup retention and on-demand purge of deleted rows (#12739)
* lance: per-table cleanup retention and on-demand purge of deleted rows

A vector deleted from an index stayed in storage until compaction
rewrote its fragment (only above 10% deletions) and the cleanup job's
168 h retention, set once per worker, aged out every version that
referenced it.

Let a table override retain_hours and min_versions in its Lance config
(seaweedfs.cleanup.retain_hours, seaweedfs.cleanup.min_versions), and
let it request a purge (seaweedfs.purge.requested). The cleanup job
proposes a table with a pending purge regardless of the version floor
and then rewrites every fragment with deletions, retrains the indices
(a remapped IVF_FLAT index still holds the deleted vector), keeps only
the latest version, and records the request as served. A version pinned
by a tag fails the purge and leaves the request pending.

The test scans every file of a local table for a deleted vector's key,
metadata and values, and finds none after the purge.

* lance: refuse to purge a sharded vector index

A retrain builds every index name over the whole table, so the purge
turned shards of one vector index (one name per fragment set) into
names that each covered every fragment, indexing every row once per
shard. Check before compacting and fail with the request left pending,
and retrain only the vector indices.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* lance: rebuild scalar indices in a purge

Compaction only remaps scalar indices, and a retrain leaves them alone,
so a purge reported success while a bitmap index still listed the
deleted row's value, and under stable row ids a BTree index still held
the deleted key. Build each scalar index again from the remaining rows
with the parameters it was built with.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* lance: purge through the cleanup job's retention policy

The purge built its own cleanup policy, so a deployment that has the
cleanup job keep more than the floor and the window (versions an
unleased reader may still be on) would have lost them to a purge. Clean
up through retention_policy, the one the job itself uses, with the
latest version kept.

Mark the request served only once the oldest version left is one the
rebuild wrote, and record the rebuild so a purge that has to wait for
its cleanup (a tag, or that policy) retries only the cleanup instead of
compacting and retraining the indices on every pass.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* vector: lease the version every list page reads

A list page read its version without a lease unless it was a segment
that read data and metadata twice. A purge removes every version but
the latest, and only a lease tag stops it, so a listing that opened the
version before the purge's compaction could lose its files mid-read.
Pin it as get_vectors and query_vectors do.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* lance: give the cleanup job a build's execution time

A purge compacts the table and retrains its indices inside a cleanup
job, but the job declared no runtime defaults, so the admin held it to
its 90 s default and could expire a purge mid-rebuild. Declare the same
budget the indices job uses for a retrain.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-18 14:24:28 -08:00
c7d8b106d5 mount: follow a peer unlink to the surviving hard link for closed inodes (#12737)
* mount: drop a peer-unlinked name only while its inode keeps another

Move the survivor check from followSurvivingLink into InodeToPath.RemoveAlias
so the check and the removal happen under one table lock.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* mount: follow a peer unlink to the surviving hard link for closed inodes

A peer deleting one name of a hard link only moved an open handle to the
surviving name. A closed inode kept the deleted name as its primary path, so
the next open by the kernel's nodeid failed with ENOENT.

Co-authored-by: skymazer <54621405+skymazer@users.noreply.github.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* mount: share the cached-entry freshness check of peer invalidations

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* mount: keep a name a local rename reclaimed before the peer unlink arrived

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* mount: keep a name a local rename or link gave the inode after an older peer delete

The cached-entry check only covers a cached parent. InodeToPath now records,
per hard-linked inode, the newest name a local rename or link gave it with
the filer stamp, and the closed-inode peer-delete path keeps that name when
the delete is older. The record lives in a side map dropped on Forget, so
InodeEntry keeps its size class.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* filer: sign rename stream responses with the filer signature

A rename response carries the stamp of the filer that applied it, so the mount
can tell which clock the stamp belongs to, as it does for create acks.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* mount: order a peer delete against a local rename or link only on one filer clock

The stamp of a local rename or link comes from the filer that applied it, and
a delete logged by another filer runs on a different clock. Keep the name only
when both carry the same nonzero filer signature; otherwise drop it as before.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* mount: drop the local-name stamp fence for uncached parents

Ordering a delayed peer delete against a local rename by filer stamp needs a
record per alias, a shared filer clock, and the open-handle path to follow;
that is a larger change than this fix. Back to the cached-entry check, which
covers a cached parent.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: skymazer <54621405+skymazer@users.noreply.github.com>
2026-11-18 14:13:40 -08:00
a3b2fd722d mount: report opens for reading with TouchAccessTime (-accessReport) (#12721)
* mount: report opens for reading with TouchAccessTime (-accessReport)

An open answered from the metadata cache never reaches the filer, so a
filer that tracks access cannot see that a cached file is still being
read. With -accessReport, the mount sends one TouchAccessTime per open
for reading, with the open time, whether the open was answered from the
cache or not. Write-only opens are not reported.

Reports go out in the background and never hold up the open: a report
that does not fit in the queue is dropped, and drops and failures are
logged once a minute. A filer that answers Unimplemented pauses
reporting for 10 minutes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* mount: address review of the access report

- Drop reports still queued when reporting pauses; the filer that answered
  Unimplemented would answer them the same way.
- Bound each filer attempt to 2s, so a filer that accepts the connection
  but never answers leaves time for the next one, and keep using the filer
  that answered.
- Fail over only when a filer cannot be reached. Any answer, Unimplemented
  included, ends the walk.
- Report an O_CREAT open of an existing file with read access, which comes
  through Create instead of Open. A new file has nothing to read and is not
  reported.
- Stop the reporter on unmount: stop ends the workers and cancels the
  report in flight.
- Say in the flag help that it has no effect unless the filer implements
  TouchAccessTime.
- Tests no longer wait a fixed time to see that nothing else was sent; they
  look at the queue of a reporter that is not started.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* mount: fail an access report over only when the filer did not answer

A status the filer sent ends the walk even when its message contains a
word the transient classifier matches, so an Unimplemented answer still
pauses reporting and a rejected request is not resent elsewhere.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* mount: keep a filer selection made while an access report failed over

A report that reached a later filer moves the mount there only if the
selection it started from is still current, so a slower report cannot
undo another call's failover.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* mount: share an access report's time with the filers not yet tried

Each attempt is capped by an equal share of the report time left, so a
row of stalled filers cannot use up the report before a healthy one later
in the list is tried.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* mount: report the path an open for reading came through

A create that opens an existing hard link reports the name it was
given rather than the inode's first known name. Open reads the inode's
path from the inode table rather than the handle's saved path, which
other handle operations may rewrite concurrently.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* mount: check the access report stop and failover tests without races

The post-stop report is checked on a reporter without workers, so no
worker can take an accepted report before the check, and the failover
test asserts that the stalled filer was tried first.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* pb: keep the shared channel when the peer answers with its own status

gRPC never raises InvalidArgument, NotFound, AlreadyExists,
FailedPrecondition, OutOfRange or DataLoss itself, so such a status is
the peer's answer. It fell through to the string matching, and a message
naming a path like /transport closed the cached channel and cancelled
every other call on it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* mount: say which filer selection an access report keeps

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* mount: bound the in-flight call in the rejected-report channel test

The call held in flight went through touchAccessTime, whose 2 s attempt
cap could expire on a slow runner and fail the test on a healthy channel.
It now uses callFiler with the test deadline. The wait for it to reach
the filer also ends if it returns early or the deadline passes, and the
blocked handler is released on every exit path.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* mount: report each file at most once per -accessReport.interval

Every read open used to send a TouchAccessTime, adding a filer metadata
call per open. A path reported within the interval (default 1h) is now
skipped; the last report times live in a bounded LRU. 0 keeps reporting
every open.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* mount: report an open again once the file was modified since its last report

Within -accessReport.interval a read open was skipped even after a write,
so the filer kept an atime older than the mtime until the interval ran
out. relatime records that first read after a write; do the same by
comparing the entry's mtime with the last report.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* mount: keep the report time apart from the mtime it saw

Recording max(now, mtime) as the last report let a file dated a day ahead
suppress reports for a day plus the interval. Record when the report was
sent and the mtime the entry had, and treat any change of that mtime as a
modification, so neither clock can stretch or shorten the interval.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: MapleWhiteZZZ <18653045+MapleWhiteZZZ@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
2026-11-18 14:11:11 -08:00
Chris LuandClaude Opus 5.5 9f413ff2c8 shell: volume.check.disk skips replicas whose live counts match (#12743)
* shell: volume.check.disk skips replicas whose live counts match

FileCount counts every write and DeleteCount every delete or overwrite
until a vacuum drops them, so a replica vacuumed while its peer was not
keeps a different FileCount forever and was re-checked on every run
although both hold the same live needles. Only treat counts as differing
when the live counts (FileCount - DeleteCount) differ too.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* shell: volume.check.disk skips equal live counts only across compaction revisions

Equal live counts at the same compaction revision mean a missed write
and a missed delete, not a vacuum-state difference, so keep checking
those pairs.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* shell: volume.check.disk real-time recheck ignores heartbeat revisions

VolumeStatus counts are newer than the heartbeat's compaction revisions,
so a vacuum in between could make a pair at the same revision look
vacuum-skewed. Apply the old count rules there.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* shell: volume.check.disk skips on live counts only when no repair could follow

Equal live counts across compaction revisions can still hide a missed
write and a missed delete. Keep checking such pairs when -syncDeleted
could propagate the delete, or when -resurrectMissingNeedles could copy
a needle into a never-vacuumed replica.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-18 14:10:47 -08:00
e037a9c1ce s3vectors: enforce AWS metadata, key and filter limits (#12738)
* s3vectors: enforce AWS metadata, key and filter limits

PutVectors accepted vector metadata of any size and key count, bounded
only by the request body limit, and keys of any length; QueryVectors
accepted filters of any size. Apply the AWS S3 Vectors limits
(https://docs.aws.amazon.com/AmazonS3/latest/userguide/s3-vectors-limitations.html),
taking the ENHANCED index values where the modes differ, since this
implementation prefilters and takes no indexMode:

- 100 KB total metadata per vector, which also bounds a single
  non-filterable field;
- 2 KB filterable metadata per vector;
- 50 metadata keys per vector;
- 1024-character vector keys;
- at most 10 nonFilterableMetadataKeys of 1 to 63 characters at
  CreateIndex;
- 100 filter constraints per QueryVectors request.

A violation is a ValidationException before the engine is called.

* s3vectors: accept metadata whose JSON is exactly at the byte limits

metadataEntrySize counts a trailing comma for every member, so starting
from both braces overcounted by one and rejected metadata that encodes to
exactly 2 KB filterable or 100 KB total.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3vectors: rewrap the metadata limits header comment

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-18 13:49:15 -08:00
Denis 3db54ee7e5 kms/openbao: refresh the AppRole token before it expires (#12734)
* kms/openbao: refresh the AppRole token before it expires

The provider logged in with AppRole once, when it was created, and
never renewed the token. After the token's TTL every SSE-KMS upload,
and every read of an SSE-KMS object whose data key was not cached,
failed with "permission denied" (HTTP 500 to the client) until the S3
server was restarted.

With AppRole authentication the provider now logs in again in the
background once two thirds of the token's TTL have passed, and retries
a failed login every 5 s. Each login is bounded by the client's
timeout, or by 30 s when request_timeout is not positive. Close stops
the refresh. A static token, or an AppRole token without a TTL, is left
as it is.

* kms: keep the OpenBao token refresh across configuration reloads

- AddKMSProvider and InitializeGlobalKMS create a provider instance of
  their own with the new NewProvider instead of the registry's cached
  one: a reload closed the cached instance and then got it back, which
  now stopped its token refresh. The new instance is created before the
  old one is closed, so a configuration that fails leaves the current
  provider working
- openbao: time each refresh from the start of the login that issued
  the token, since its TTL runs from then rather than from the response
- tests: a reload gives a new instance and closes the old one, a failed
  reload keeps the old one; the refresh survives a reload; a slow login
  response

* kms: create a replacement provider outside the manager lock

AddKMSProvider ran the provider factory, which for OpenBao is a login
that may wait for its timeout, while holding the manager lock, so every
lookup, also of unrelated providers, waited for it. The new provider is
now created before the lock is taken; the lock covers only the swap, and
the replaced provider is closed after it. Test: a lookup while another
provider is being created.
2026-11-18 11:07:09 -08:00
Chris LuandClaude Opus 5.5 94de65b3b5 s3api: answer a malformed Cache-Control on PutObject with InvalidRequest (#12733)
PutObject rejected a Cache-Control value it could not parse with
InvalidDigest, whose message is about Content-MD5, even when the request
sent no Content-MD5. CreateMultipartUpload already answers the same parse
error with InvalidRequest; PutObject now does too.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-18 11:03:57 -08:00
ebc77400ca operation: reassign an upload after a proven-expired volume JWT (#12729)
* operation: reassign an upload after a proven-expired volume JWT

* seaweed-volume: mark an expired write JWT refused before append

Mirror the Go volume server: a multipart POST whose write JWT is signed
with the current key, names this fid and fails only on exp gets
X-Seaweed-Write-Outcome: expired-jwt-before-append on its 401, so the
uploader reassigns instead of failing. Both checks use one guard
snapshot, so a key rotation between them cannot drop the marker.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* operation: keep same-FID retry for filer-proxy expiries, snapshot the write signing key, and make the RetryForever budget test fail fast

* operation: re-mint the filer credential before a proxy same-FID expiry retry

* operation: make a proxy marked-expiry terminal in the outer retry too, not only the inner same-FID loop

---------

Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-18 11:02:52 -08:00
Chris LuandClaude Opus 5.5 ecbf7af1ca mount: compare permissions against local ids without mapping them again (#12726)
* mount: compare permissions against local ids without mapping them again

Access, open and create (without -defaultPermissions) ran FilerToLocal on
entries that maybeReadEntry and maybeLoadEntry already return in local ids.
Under a uid/gid map where a local id is also a mapped filer id, the owner
was mapped twice and the check refused the file's own owner.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* mount: cache a deferred create's placeholder in filer ids

The meta cache stores filer uid/gid and maps them to local ids on every
read, but createRegularFile cached the deferred or writeback placeholder
with the caller's local ids. The second mapping in the permission checks
used to cancel that out under a swapping id map; without it the owner of
a placeholder was refused. Cache a mapped clone and keep the handle's
entry in local ids.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* mount: test that a deferred create returns its entry in local ids

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-18 11:01:30 -08:00
b64895ed0a filer: stop following a peer that stays unreachable (#12725)
* filer: stop following a peer that stays unreachable

* filer: re-track an answering peer only while it is still followed

A signature read that succeeds just before the master removes the peer
called initPeerWatermark after the removal, clearing its mark. The loop
then exits, so the watermark it re-tracked never advances or drops and
pins aggregated readers at the settled horizon.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* filer: forget a peer's unreachable clock when the master removes it

Only a successful connection cleared the clock, so every departed address
that once failed stayed in the map for the filer's lifetime, and a re-added
peer inherited the old outage. The removal now drops it, and the clock is
guarded by peerChansLock so the removed peer's exiting loop cannot start a
new one.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* filer: report the unreachable mark only when it is set

Once peerResumeRetention forgets the mark of a peer that stays
unreachable, markPeerWatermarkRemoved no longer marks it (its watermarks
are gone), but notePeerUnreachable still saw no mark and logged
"treating it as removed" on every retry. markPeerWatermarkRemoved now
reports whether it set a new mark.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test: fix the unreachable-peer subscriber's start before the baseline write

The subscriber goroutine picked its start time itself, so a baseline write
that finished first fell before the subscription and the test failed after
20 seconds with correct delivery.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-18 11:00:15 -08:00
72ac9e96b2 mount: answer an interrupted read with EINTR, not EIO (#12723)
* mount: answer an interrupted read with EINTR, not EIO

The kernel's FUSE contract (Documentation/filesystems/fuse/fuse.rst,
"Interrupting filesystem operations") lets a filesystem either ignore an
INTERRUPT or honor it by replying EINTR to the original request. Read did
neither: the FUSE cancel channel cancels the read's context, the chunk
fetch fails with "context canceled", and every error became EIO.

Interrupts come from any signal the reader handles. Go's runtime sends
SIGURG for goroutine preemption, so Go programs reading under load hit
this on healthy data: on a 128-thread 64 KiB random-read test about 250
of 5 million reads failed with EIO, and none with GODEBUG=asyncpreemptoff=1.

Read now returns EINTR when its own context was cancelled, the way
dirReadStatus already does for directory reads and lookups do since
#11717. Go's os.File retries EINTR, so those readers see no error. A
cancelled read also skips the entry refresh retry, and neither read log
line reports an interrupt as a warning or error. Failures without an
interrupt stay EIO.

Tests: TestReadInterruptedReturnsEINTR (fails on the old code with EIO),
TestReadFailureWithoutInterruptReturnsEIO; weed/mount/... passes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* mount: require the failed-read test to reach the filer

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* mount: skip the read retry once the request is interrupted

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
2026-11-18 10:58:05 -08:00
11af0e0c12 mount: move an open file's read view to the chunks a merge wrote (#12722)
* mount: move an open file's read view to the chunks a merge wrote

When a flush finds that a file's chunks take more than twice its size,
maybeMergeChunks reads the file back and writes it again as new chunks.
The commit that publishes them makes the filer delete the chunks they
replaced. The flush put the new chunks into the handle's entry but left
its read view on the old ones, so every descriptor that kept the file
open went on reading the replaced chunks. Those read back until their
volume is vacuumed. After that a read first retries the fetch of the
missing needle for about 13 s; then refreshEntryForRead installs the
filer's entry and the read is retried. The refresh skips a handle with
writes it has not flushed, and such a read returns EIO instead.

Files that are kept open and rewritten in place, such as PostgreSQL
data files and SQLite databases, are the ones whose chunks bloat, so
they are the ones that hit this.

installRewrittenChunks puts the merge's chunks into the entry and the
read view together, under the entry lock. Chunks attached while the
merge ran were not part of it and stay. Flushes without a merge are
unchanged.

TestMergeMovesReadViewOffReplacedChunks covers the handle.
TestReadOpenHandleAfterChunkMerge runs it on a real mount, with and
without an unflushed write; for it, the fuse_integration framework
starts mini on a free master port and exposes the address.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* mount: drain uploads before a merge replaces the chunks

Uploads attach their chunks without the handle lock, first to the entry
and then to the read view, and writes that landed after doFlush drained
the pages may still be uploading when the flush takes its lock. An
upload that attached during the merge could keep a chunk older than
data the merge already holds, or put a chunk the merge replaced back
into the read view. Either outranks the merge's zero-timestamp chunk.

When the chunks are bloated enough to merge, the flush now drains the
pages again while its lock holds writes off, and takes the chunks again.
FlushAll returns only after every upload has attached to both the entry
and the read view, so nothing attaches during the merge.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* mount: leave a pre-merge upload failure to the next flush

The drain before a merge returned an upload failure through the
metadata flush. doFlush then retried it as a filer error and answered
EIO, where the page flush answers a full disk with ENOSPC. The drain is
there only to wait for the uploads: a failure stays with the pages, and
the next flush reports it as a write error, as it would without a merge.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* mount: keep chunks attached while a flush installs its compacted list

A write that lands after doFlush drained the pages can still be uploading
when flushMetadataToFiler takes its chunk snapshot. Its upload attaches the
chunk without the handle lock, and the plain flush then replaced the
entry's chunks with the list compacted from the snapshot, dropping it from
the handle and from every later commit. Keep chunks attached after the
snapshot, as the merge path already does.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* mount: fail a merge's flush on a failed drain

A failed upload found by the drain before a merge was only logged, and
the flush went on to publish a size covering bytes that no chunk stores.
It now fails the flush as a pageUploadError before anything is
published; doFlush maps it with writeErrorToFuseStatus, as it does its
own FlushData failure, and does not retry it as a filer error.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* mount: clone a flush's request entry under the entry lock

The request snapshot cloned the live entry after GetEntry had released
its read lock. Uploads attach chunks without the handle lock, so an
AppendChunks that landed during the clone raced it on the Chunks field.
Clone inside UpdateEntry, as content flush recovery does.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* mount: leave a merge to a later flush while uploads are pending

Before a merge the flush drained the pages while it held the handle and
publication locks. A writeback upload retries a transient storage error
until it succeeds, so that drain could hold both locks for as long as
storage stayed down, stalling reads, writes and stat on the handle.
The flush now checks for pending uploads and, if any, publishes the
compacted chunks and leaves the merge to a later flush. With none
pending the lock still holds writes off, so nothing attaches during the
merge, and a failed upload still fails the flush as a write error.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* mount: keep room for pending uploads when a flush merges onto a moved entry

A flush that leaves the chunk merge to a later flush publishes while
uploads are still pending, and writes that landed after the drain can be
uploading on any flush. Merging a peer's inline content or a shrink into
such a publication lets the upload attach its chunk next to the inline
bytes, where reads never reach it. Tell createContentEntry when uploads
are pending so the merge keeps room for them.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* mount: test chunk merges through the filer's chunk proxy, also under load

The tests serve the filer's chunk proxy, and like the filer the fake
deletes the chunks each commit replaces, so a read view still naming
them fails:
- TestFlushMergeThroughChunkProxy/Quiet: the flush merges, and reads
  through the open handle return the file from the merged chunk alone.
- TestFlushMergeThroughChunkProxy/UploadInFlight: an older page is still
  uploading while a newer page of the same bytes has attached. The flush
  leaves the merge, reads return the newer page, and the next flush
  merges.
- TestFlushMergeRacesWithWritesAndReads: four writers, three flushers
  and a reader on one handle, with uploads delayed at random. The
  writers pause now and then, so 14 to 18 flushes a run merge and the
  others leave the merge to a later one. Every read must succeed within
  2 s, and at the end the handle and the filer must both hold exactly
  what was written.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* mount: let a held upload go when UploadInFlight fails

The subtest's proxy holds the older page's upload until the test
releases it, and only the passing path did. A failure in between left
the handler blocked, so the proxy's cleanup waited on it and the failure
surfaced as a suite timeout. Release it from a cleanup as well, once,
and bound the waits for the upload and the drains.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
2026-11-18 10:51:31 -08:00
de5b3305a7 mount: ask the filer again when a deferred create is published mid-lookup (#12720)
* mount: ask the filer again when a deferred create is published mid-lookup

A program that writes a file under a temporary name and renames it while
the file is still open (safetensors save_file does, through Rust
tempfile's persist) looks the temporary name up before the rename. When
the parent directory's listing is not cached, that lookup goes to the
filer. If a FLUSH publishes the deferred create while the lookup is in
flight (a forked child closing its inherited descriptor sends one), the
filer can answer not-found from a read taken before the create
committed. By the time the answer arrives the flush has cleared the
handle's dirty flag, so lookupEntry finds nothing local to explain the
answer, logs "possible coherence bug" and returns ENOENT, and the rename
fails although the file exists.

Note before sending the request whether the inode had unpublished
metadata (a dirty handle or an async flush in flight). On not-found, if
it had and has since been published, ask the filer once more: the filer
acknowledged the publish before the second request is sent. A second
not-found stands, so a file another mount removed is not brought back.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* mount: judge a placeholder's not-found by one look at its base

Review of the previous commit found two problems. The retry check and the
not-found fallback read the handle separately, so a flush that finished
between the two reads still turned the answer into ENOENT. And the
pre-request read of dirtyMetadata, a plain bool that Write and the flush
set under the handle lock, raced with them on every lookup that reached
the filer.

Note instead whether the handle is a local placeholder: a deferred create
has no filer base (baseEntry) until a flush publishes it. A not-found for
a placeholder is judged by one atomic look at that base. With none, the
filer cannot have the file yet and the handle answers. With a base, the
filer acknowledged the publish, so ask once more. dirtyMetadata is not
read for this, and the not-found fallback keeps the existing check, with
the placeholder decision added.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* mount: answer a placeholder from its own handle; a second miss is final

Follow-ups from review of the previous commit:

- Pick the placeholder after the async-flush wait, from the path's
  current mapping.
- A placeholder that still has no base answers from the observed handle
  itself, instead of through the not-found fallback, which re-resolved
  the path and could miss once the handle was released and the cache
  entry evicted.
- A not-found from the second request is final. The flush installs the
  base before it clears dirtyMetadata, and keeps it set when a write
  landed meanwhile, so the fallback could return a file another mount had
  removed.
- Test gates release on cleanup, so a failing test leaves no handler
  blocked.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* mount: read a placeholder's entry the way GetAttr does

The previous commit answered a not-found for a placeholder from the
handle's live entry under entryLock. Neither of the entry's writers takes
that lock: Write and the metadata flush change size, times and chunks
under the handle lock, and async upload workers append chunks under the
LockedEntry lock, so the lookup could read an entry while it changed.

Answer with the copy the create cached when it is still this file's
(same inode), as the not-found fallback already prefers. Otherwise copy
the handle's entry under both writers' locks, in GetAttr's order (handle
lock shared, then the LockedEntry lock), and convert the copy. No caller
of lookupEntry holds a handle lock.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* mount: cache a deferred create's placeholder in filer ids

The meta cache stores filer uid/gid and maps them to local ids on every
read, but createRegularFile cached the placeholder with the caller's local
ids. Under a uid/gid map whose local id is also a mapped filer id, a lookup
or stat answered from the placeholder reported the wrong owner. Cache a
mapped clone and keep the handle's entry in local ids.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* mount: assert the placeholder is cached before checking its owner

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* mount: check the snapshot owner after evicting the cached placeholder

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
2026-11-18 10:50:28 -08:00
04e135c21e fix(helm): separate volume DNS discovery from ingress traffic (#12719)
* helm: set publishNotReadyAddresses on the volume headless service

* fix(helm): separate volume DNS discovery from ingress traffic

* test(helm): cover volume DNS discovery and ingress routing

* fix(helm): rename the volume routing Service so volumes.client cannot collide

* docs(helm): name the ready-only Service for explicit volume backendRefs

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(helm): fail when the volume routing Service name collides after truncation

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* docs(helm): point explicit volume backendRefs at the rendered client-volume name

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-18 10:49:35 -08:00
Chris LuandClaude Opus 5.5 862913d199 volume, ec_balance: clean up failed EC shard copies; stop re-proposing empty-index volumes (#12718)
* volume: remove files a failed VolumeEcShardsCopy wrote

The shard files were copied before the .ecx, so a source holding a
0-byte index failed the request only after its shards had landed, and
nothing removed them: every retried move stranded another unmounted
shard on a new disk.

Copy the .ecx first, staged beside its target, so a bad index fails
before any shard lands; rename it into place only after the shards and
sidecars have copied, so a failed copy never truncates or replaces an
.ecx the destination already has. On any error, remove exactly the
files this call created; files already on the disk (an existing
holder's shards, index or journal) are left alone. Go and Rust volume
servers behave the same.

Claude-Session: https://claude.ai/code/session_011UpvafcfUuC2mLxNAXLQJi

* ec_balance: skip EC volumes whose every holder reports an empty .ecx

A shard copy refuses a 0-byte .ecx, so detection proposed the same
failing move of such a volume every cycle. Leave volumes with no
holder reporting a needle out of planning; their shards still count
toward disk occupancy.

Claude-Session: https://claude.ai/code/session_011UpvafcfUuC2mLxNAXLQJi

* volume: serialize VolumeEcShardsCopy per volume

Overlapping copies of one vid shared the staged .ecx, and a failed copy's
cleanup removed shards that a concurrent copy had already committed and
mounted. Hold a per-vid EC lifecycle lock across the whole copy and its
cleanup, named as in the enterprise build (Store.EcLifecycleLock /
ec_lifecycle_lock).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* volume: overwrite an existing .ecx in place when committing a copied index

A mounted EcVolume reads its index through an open handle, so renaming
the staged copy over it left the reader on the replaced inode: tombstones
the source had folded into its .ecx never reached the mounted holder.
An index already on the disk is now overwritten in place, as before the
staging change; a fresh destination still gets the rename. Go and Rust.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* volume: stage replacements of shards a VolumeEcShardsCopy finds on disk

The shard transfer opened an existing destination shard with O_TRUNC and
removed it on a stream or write error, so a failed retry deleted a shard
this disk already held. A shard already present is now copied to
<shard>.copying and renamed into place only after every other file has
copied; a mounted copy is reopened so it serves the new file. Fresh
shards still write in place and are removed on failure. Go and Rust.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* volume: stage the .vif and .ecsum a VolumeEcShardsCopy brings until commit

The sidecars landed as soon as they transferred, ahead of the deferred
.ecx commit, so a failure after them left a new .vif beside the index
the failed copy kept. They are now staged as .copying and renamed into
place with the shards and index once every transfer has succeeded; a
source without one still stages nothing. Go and Rust.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* volume: run a Rust VolumeEcShardsCopy to completion when its caller goes away

tonic drops a cancelled handler future at its current await, which
released the EC lifecycle lock while file operations the copy had started
(a cleanup remove_file on the blocking pool, a create or rename) could
still land; a retry taking the lock could then lose its files to them.
The copy now runs in a spawned task that owns the lock, as Go's handler
runs to completion.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* volume: fail the sidecar copy tests with an unreadable source .ecsum

A directory at the source path reads as "not found" on volume servers
that skip non-regular EC files, which turned the failure into a no-op.
A mode-000 file fails the transfer on every implementation; root and
Windows, which can read it, skip the test.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* volume: copy each shard a VolumeEcShardsCopy lists once

A shard listed twice and already held here was staged twice; the first
commit rename consumed the staged file, so the second failed the request
after the shard was replaced, and every retry failed the same way.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* volume: wait out an in-flight VolumeEcShardsCopy before mounting its shards

A shard the copy had already landed could be mounted while the copy was
still running; the copy's failure cleanup then removed the file but left
the shard registered and announced. MountEcShards takes the per-volume
lifecycle lock, as the enterprise build already does, and the Rust mount
handler takes the same lock.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* volume: fail a VolumeEcShardsCopy whose replaced mounted shard cannot be reopened

The copy reported success while the mounted shard kept serving the
replaced file. The error lets the caller retry the copy instead of
treating the shard as delivered.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* volume: check which transfer fails in the EC copy cleanup tests

An earlier failure, before any shard or sidecar landed, would leave the
destination unchanged and pass every assertion without exercising the
cleanup. The sidecar test also skips on Windows, where a mode-000 file
stays readable.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* volume: fail the sidecar transfer in the EC copy test by injection

A mode-000 file is still readable to root and on Windows, so the test
skipped there. The source now returns an error for the .ecsum request
only, and the test checks that it served .vif before that failure.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* volume: take the EC lifecycle lock for a blanket full teardown

A teardown acknowledged while a detached copy of the same volume was
still waiting on its source let that copy recreate the index and shards
afterwards. Go and Rust now wait for the copy, as the enterprise build
does.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* volume: keep the request id on a Rust EC shard copy

The copy now runs in a spawned task, outside the gRPC layer's request id
scope, so its CopyFile calls to the source went out with a new id. Carry
the caller's id into the task.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* volume: unmount a replaced EC shard whose reopen fails

The failed copy left the mount reading the replaced file's inode, so the
destination kept serving the old shard. Unmount it, which also reports
the removal to the master, so a retry copies and mounts it again.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* volume: make the Rust EC copy tests fail for the right reason

The sidecar test now serves a directory as the source .ecsum, which root
cannot read either, so it no longer returns early there. The cancel test
holds the store until the handler's join reports the cancellation, so a
copy that finished before the abort can no longer pass it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* volume: stage every shard a VolumeEcShardsCopy transfers

A shard missing here was written in place with a cleanup guard armed
before its transfer started, so a failed copy removed a shard that
ReceiveFile wrote meanwhile. Every shard is now staged, and only one
that did not exist when it is moved into place is removed on failure.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* volume: take the EC lifecycle lock for VolumeEcShardsRebuild

A rebuild folds the deletion journal into .ecx. Run during a copy of the
same volume, the copy's index then replaced the folded tombstones. Go
and Rust take the lock the enterprise build already takes here.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* volume: check the staged .vif bytes in the EC sidecar failure test

A source without the optional .vif also returns success, so the served
extension list could not prove the transfer. The test now reads the
staged .vif when the .ecsum request fails and compares it with the
source's.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* volume: unmount a failed reopen's shard on the copy's disk only

UnmountEcShards walks every disk, so a reopen failure on the destination
also unmounted a healthy copy of the same shard on a sibling disk. The
copy now unloads the shard from its own disk and reports that disk's
removal.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* volume: take the EC lifecycle lock for an ordinary EC shard delete

Shards a copy has staged are invisible to the delete's last-shard check,
so deleting the only canonical shard removed the index the copy was
about to commit its shards against. Go and Rust deletes now wait for an
in-flight copy of the same volume.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* volume: leave a mounted shard's copied file in place on copy failure

A shard mounted on the copy's disk whose file was missing got the copied
file reopened into its mount, then that file was unlinked by a later
failure's cleanup while the mount kept serving it. Only an unmounted
shard's new file is removed now.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* volume: skip repeated shard ids with a seen set

Scanning the request prefix for each id made a request with many
repeats quadratic while the copy held the volume's lifecycle lock.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* volume: make the Go EC copy and lock tests observe their effects

The fake source counts requests per extension, so the duplicate-id test
proves .ec05 is fetched once, and returns fixture errors instead of
failing the test from its handler goroutine. The staged .vif reaches
the test through a channel. The mount and teardown lock tests seed real
files and check that nothing happened while the lock was held and that
the operation took effect after it was released.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* volume: poll the handler directly in the Rust EC lifecycle lock tests

A spawned handler that had not run when the 200 ms timer fired passed
the blocking check without ever reaching the lock. The tests now poll
the pinned handler future themselves, which guarantees it is waiting on
the lock, and cover the ordinary shard delete as well.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* volume: observe the lifecycle-lock waiter in EC copy lock tests

The mount, teardown, shard-delete and rebuild tests slept 200 ms while
holding the lock, so a handler scheduled only after the unlock passed
without ever taking it. Wait until the handler's goroutine is blocked on
the mutex instead; each test now fails within its timeout if the handler
skips the lock.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* volume: check the shard is staged before the failure-path write in a copy test

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* volume: recover a missing EC index under the volume's lifecycle lock

Missing-index recovery fetched the index and then ran a store-wide mount
of every orphan shard, without the per-volume lifecycle lock. It could
mount shards an in-flight VolumeEcShardsCopy of another volume had just
renamed in, and that copy's failure cleanup then unlinked a mounted shard.
Each recovered volume now holds its lifecycle lock across a re-check, the
fetch and a mount restricted to that volume.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* volume(rs): recover a missing EC index under the volume's lifecycle lock

Mirror of the Go change: recover_missing_ec_indexes takes the volume's
ec_lifecycle_lock, re-checks that the index is still missing, fetches it,
and mounts only that volume's shards through mount_recovered_ec_shards_only,
so an in-flight VolumeEcShardsCopy of another volume never has its
renamed-in shards mounted from under it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* volume: mount a volume whose index a copy installed during recovery

Recovery re-checks under the lifecycle lock that the index is still
missing. When a copy installed it in the meantime, the re-check skipped
the mount too, and the volume's other shards stayed unmounted because
later scans no longer list it. Skip only the fetch and still mount the
volume's orphan shards under the lock, in Go and Rust.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* volume: fetch a recovered EC index into the disk the locked re-check found

The re-check under the lifecycle lock only kept a yes/no answer, so the
fetch still wrote to the destination from the first scan, which may be a
disk that has since gone away. Fetch into the record the re-check returned.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* volume: mount a copy-installed EC index even when no peer is known

Recovery skipped the locked step when the master was unreachable, the
lookup failed or no peer was listed. A copy that installed the index after
discovery then left the volume's other shards unmounted, and later scans
no longer list it. Always run the locked re-check, and give up for lack of
peers only when the index is still missing.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* volume: grow the stack buffer before scanning for a lock waiter

A full buffer can cut off the handler goroutine's stack, so the test would
report a handler that is waiting as one that never waited.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* volume(rs): fetch a recovered EC index into the disk the locked re-check found

Mirror of the Go change: fetch into the record the re-check under the
lifecycle lock returned instead of the destination from the first scan.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* volume(rs): mount a copy-installed EC index even when no peer is known

Mirror of the Go change: a failed lookup or an empty peer list no longer
skips the locked re-check, so a volume whose index a copy installed after
discovery still gets its orphan shards mounted.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* volume(rs): run missing-index recovery to completion when the mount RPC is cancelled

Dropping the mount handler mid-recovery released the volume's lifecycle
lock while the .ecj merge it had started on a blocking thread was still
landing, so a teardown could remove the volume before that late write.
Run recovery on its own task, as the copy handler does, so the lock is
held until it finishes; Go's handler already runs it to completion.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* volume: skip other volumes before stat in a single-volume EC scan

Recovery re-checks and mounts each volume with its own scans, so a
request with many orphans repeated whole-store discovery once per volume.
The index-owner and orphan-shard scans now take the same volume filter
and drop other volumes' files by name, before the per-file stat, and the
locked re-check scans only its own volume.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* volume(rs): skip other volumes before stat in a single-volume EC scan

Mirror of the Go change: the index-owner and orphan-shard scans take the
volume filter and drop other volumes' files by name before the per-file
stat, so recovery's per-volume re-check and mount no longer repeat
whole-store discovery under the store lock for every orphan.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* volume(rs): announce shards recovered after the mount RPC was cancelled

Recovery now outlives a cancelled mount RPC, but the heartbeat nudge
stayed in the handler, so shards it mounted after the caller left waited
for the next full EC heartbeat. Nudge it from the recovery task, as Go's
loads announce each recovered shard.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* volume: match the collection as well as the volume id in the recovery re-check

Discovery keys orphans by (collection, vid), but the locked re-check
matched the vid only. Two collections with orphans of the same vid could
both resolve to the first record, so the other's index was never fetched.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* volume(rs): match the collection as well as the volume id in the recovery re-check

Mirror of the Go change.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-18 10:48:21 -08:00
Chris LuandClaude Opus 5.5 8c6d09f498 s3api: answer NoSuchBucket from GetBucketRequestPayment for a missing bucket (#12716)
* s3api: answer NoSuchBucket from GetBucketRequestPayment for a missing bucket

The handler wrote the static BucketOwner answer without resolving the
bucket, so a bucket that does not exist looked like one that does.
Check the bucket first, as PutBucketRequestPayment and the other static
configuration getters already do.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3api: test GetBucketRequestPayment on existing and missing buckets

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-17 22:38:48 -08:00
eb1dca8088 filer: release the stamps of parent directories a rename creates (#12711)
* filer: release the stamps of parent directories a rename creates

* filer: test that a suppressed create keeps the moved entry's stamp in flight

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-17 22:37:56 -08:00
deacd575ea operation: prepare upload content before assigning the volume (#12714)
* operation: prepare upload content before assigning the volume

* operation: keep the existing upload function names for prepared payloads

UploadData and doUpload prepare once and hand the prepared payload to
retriedUploadData, so the retry loop, doUploadData, uploadChunk and
uploadChunkToHolders take a preparedUpload directly instead of growing
a prepared twin of each and branching on nil.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* operation: send ciphertext whenever the prepared payload carries a cipher key

doUploadData chose the cipher branch from the option, so a payload
prepared under a different option could be sent as plaintext metadata
or lose its key. Decide from the prepared payload itself.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* operation: build a chunk's preparation and upload options in one place

The chunk loop prepared with a hand-written option while uploadChunk and
uploadChunkToHolders built their own, so a setting that shapes the bytes
could reach one and not the other. chunkUploadOption now feeds all three.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* operation: test that a reassigned chunk reuses its prepared payload

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-17 22:36:46 -08:00
Chris LuandClaude Opus 5.5 194e6ede9f seaweed-volume: reject FetchAndWriteNeedle reads that miss the requested size (#12717)
The Go volume server fails the RPC when the remote read returns a
different length than req.Size; the Rust server cached whatever came
back as a full-size chunk.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-17 22:35:38 -08:00
475 changed files with 42096 additions and 4277 deletions

No files matched your search

+92 -21
View File
@@ -22,6 +22,16 @@ CHART = ROOT / "k8s/charts/seaweedfs"
SECTIONS = ("jwt.signing", "jwt.signing.read", "jwt.filer_signing", "jwt.filer_signing.read")
KEYS = {section: f"active-{index}" for index, section in enumerate(SECTIONS)}
ESCAPED_HEADER = '["jw\\u0074".signing]\nkey = "existing"'
# No workload is needed to exercise Helm's real lookup and update.
RELEASE_SETTINGS = (
"master.enabled=false", "volume.enabled=false", "filer.enabled=false",
"global.seaweedfs.createClusterRole=false",
"global.seaweedfs.securityConfig.jwtSigning.volumeWrite=true",
"global.seaweedfs.securityConfig.jwtSigning.volumeRead=true",
"global.seaweedfs.securityConfig.jwtSigning.filerWrite=true",
"global.seaweedfs.securityConfig.jwtSigning.filerRead=true",
)
CONFIGMAP_CHART = ("seaweedfs", "--repo", "https://seaweedfs.github.io/seaweedfs/helm", "--version", "4.54.0")
def run(*args):
@@ -77,8 +87,7 @@ def fixtures():
def check_generated(value):
decoded = base64.b64decode(value, validate=True).decode("ascii")
assert len(decoded) == 10 and decoded.isascii() and decoded.isalnum(), "invalid generated JWT key"
assert len(value) == 64 and value.isascii() and value.isalnum(), "invalid generated JWT key"
def check_helpers(helm, reference_helm):
@@ -147,20 +156,11 @@ def check_upgrades(helm, context):
legacy = "seaweedfs-security-config"
kubectl = ["kubectl", "--context", context, "-n", namespace]
release_args = ["jk", str(CHART), "--kube-context", context, "-n", namespace]
# No workload is needed to exercise Helm's real ConfigMap lookup and update.
for setting in (
"master.enabled=false", "volume.enabled=false", "filer.enabled=false",
"global.seaweedfs.createClusterRole=false",
"global.seaweedfs.securityConfig.jwtSigning.volumeWrite=true",
"global.seaweedfs.securityConfig.jwtSigning.volumeRead=true",
"global.seaweedfs.securityConfig.jwtSigning.filerWrite=true",
"global.seaweedfs.securityConfig.jwtSigning.filerRead=true",
):
for setting in RELEASE_SETTINGS:
release_args += ["--set", setting]
def stored():
cm = json.loads(run(*kubectl, "get", "configmap", current, "-o", "json"))
return keys(cm["data"]["security.toml"])
return secret_keys(kubectl, current)
def upgrade():
run(helm, "upgrade", *release_args)
@@ -168,8 +168,8 @@ def check_upgrades(helm, context):
def patch(raw):
# Seed previous-release content without taking Helm 4's SSA ownership.
run(*kubectl, "patch", "configmap", current, "--type=merge", "--field-manager=helm", "-p",
json.dumps({"data": {"security.toml": raw}}))
run(*kubectl, "patch", "secret", current, "--type=merge", "--field-manager=helm", "-p",
json.dumps({"stringData": {"security.toml": raw}}))
run(*kubectl, "create", "namespace", namespace)
try:
@@ -194,17 +194,24 @@ def check_upgrades(helm, context):
assert upgrade() == actual, f"{name}: subsequent upgrade changed a key"
print(f"PASS upgrade: {name}")
# Migration from the old chart name, followed by precedence of the current name.
# Migration from the ConfigMaps of earlier chart versions, then precedence of the Secret.
configmap_raw = "\n".join(f'[{section}]\nkey = "configmap-{index}"'
for index, section in enumerate(SECTIONS))
run(*kubectl, "create", "configmap", current, "--from-literal=security.toml=" + configmap_raw)
run(*kubectl, "delete", "secret", current)
assert upgrade() == keys(configmap_raw), "ConfigMap keys were not preserved"
print("PASS upgrade: ConfigMap migration")
legacy_raw = "\n".join(f'[{section}]\nkey = "legacy-{index}"'
for index, section in enumerate(SECTIONS))
run(*kubectl, "create", "configmap", legacy, "--from-literal=security.toml=" + legacy_raw)
run(*kubectl, "delete", "configmap", current)
run(*kubectl, "create", "configmap", legacy, "--from-literal=security.toml=" + legacy_raw)
run(*kubectl, "delete", "secret", current)
assert upgrade() == keys(legacy_raw), "legacy ConfigMap keys were not preserved"
print("PASS upgrade: legacy ConfigMap migration")
current_raw = next(fixtures())[1]
patch(current_raw)
assert upgrade() == keys(current_raw), "legacy ConfigMap overrode current ConfigMap"
print("PASS upgrade: current ConfigMap takes precedence")
assert upgrade() == keys(current_raw), "legacy ConfigMap overrode the Secret"
print("PASS upgrade: Secret takes precedence")
patch(ESCAPED_HEADER)
try:
@@ -213,13 +220,76 @@ def check_upgrades(helm, context):
assert "unsupported quoted section header" in error.stderr, error.stderr
else:
raise AssertionError("unsupported quoted header silently rotated its key")
assert keys(run(*kubectl, "get", "configmap", current, "-o",
"jsonpath={.data.security\\.toml}"))["jwt.signing"] == "existing"
assert stored()["jwt.signing"] == "existing"
print("PASS upgrade: unsupported quoted header leaves stored key untouched")
finally:
run(*kubectl, "delete", "namespace", namespace, "--wait=false")
def secret_keys(kubectl, name):
secret = json.loads(run(*kubectl, "get", "secret", name, "-o", "json"))
return keys(base64.b64decode(secret["data"]["security.toml"]).decode())
def check_configmap_release_upgrade(helm, context):
namespace = "jwt-key-migrate-" + uuid.uuid4().hex[:8]
name = "jk-seaweedfs-security-config"
kubectl = ["kubectl", "--context", context, "-n", namespace]
common = ["--kube-context", context, "-n", namespace]
for setting in RELEASE_SETTINGS:
common += ["--set", setting]
upgrade = ["upgrade", "jk", str(CHART), *common, "--set", "master.enabled=true"]
def refuse_partitioned(stage):
try:
run(helm, *upgrade, "--set", "master.updatePartition=1")
except subprocess.CalledProcessError as error:
assert "updatePartition=0" in error.stderr, error.stderr
else:
raise AssertionError(f"partitioned upgrade removed the ConfigMap ({stage})")
def exists(kind):
return run(*kubectl, "get", kind, name, "--ignore-not-found", "-o", "name").strip()
run(*kubectl, "create", "namespace", namespace)
try:
run(helm, "install", "jk", *CONFIGMAP_CHART, *common)
before = keys(run(*kubectl, "get", "configmap", name, "-o", "jsonpath={.data.security\\.toml}"))
assert set(before) == set(SECTIONS), "ConfigMap chart omitted a JWT section"
refuse_partitioned("ConfigMap release")
print("PASS upgrade: partitioned upgrade from the ConfigMap chart is refused")
# Fail an upgrade after Helm created the Secret, so it never removes the ConfigMap.
run(*kubectl, "create", "quota", "no-statefulsets", "--hard=count/statefulsets.apps=0")
try:
run(helm, *upgrade)
except subprocess.CalledProcessError as error:
assert "quota" in error.stderr, error.stderr
else:
raise AssertionError("upgrade was not blocked by the quota")
run(*kubectl, "delete", "quota", "no-statefulsets")
assert exists("secret") and exists("configmap"), "failed upgrade did not leave both objects"
refuse_partitioned("retry after a failed upgrade")
print("PASS upgrade: partitioned retry after a failed migration is refused")
run(helm, "upgrade", "jk", str(CHART), *common)
assert secret_keys(kubectl, name) == before, "upgrade from the ConfigMap chart changed a key"
assert not exists("configmap"), "upgrade left the ConfigMap behind"
print("PASS upgrade: ConfigMap chart release moves its keys to the Secret")
# The unprefixed legacy name can belong to another release in the namespace.
other = "seaweedfs-security-config"
run(*kubectl, "create", "configmap", other, '--from-literal=security.toml=[jwt.signing]\nkey = "other"')
run(*kubectl, "annotate", "configmap", other, "meta.helm.sh/release-name=other",
"meta.helm.sh/release-namespace=" + namespace)
run(helm, *upgrade, "--set", "master.updatePartition=1")
assert secret_keys(kubectl, name) == before, "another release's ConfigMap changed a key"
assert run(*kubectl, "get", "configmap", other, "-o", "name").strip(), "removed another release's ConfigMap"
print("PASS upgrade: another release's legacy ConfigMap does not block a partitioned upgrade")
finally:
run(*kubectl, "delete", "namespace", namespace, "--wait=false")
def main():
parser = argparse.ArgumentParser(description=__doc__)
parser.add_argument("--helm", default="helm")
@@ -230,6 +300,7 @@ def main():
check_helpers(args.helm, args.reference_helm)
if args.context:
check_upgrades(args.helm, args.context)
check_configmap_release_upgrade(args.helm, args.context)
if __name__ == "__main__":
@@ -1,307 +0,0 @@
#!/usr/bin/env python3
"""Regroup a GitHub release's generated notes by component.
GitHub's generated notes are one flat list of PRs in merge order. This keeps every
line as GitHub wrote it, but sorts the PRs under component headings (S3 API, Filer,
Mount, Volume Server, ...), with dependency bumps folded at the end.
The component comes from the PR title prefix, which nearly every PR has
("s3api: ...", "fix(mount): ...", "filer.sync: ...", "rust volume: ..."). A PR with
no recognised prefix is placed by the files it changes.
Usage:
release_notes_by_component.py --tag 4.54 # print the regrouped notes
release_notes_by_component.py --tag 4.54 --write # also update the release
Running it again on notes it already regrouped gives the same result.
"""
import argparse
import json
import re
import subprocess
import sys
# Display order. Each PR lands in exactly one section.
SECTIONS = [
"S3 API",
"S3 Tables and Iceberg",
"Vectors",
"Filer",
"Mount",
"Volume Server",
"Rust Volume Server",
"Erasure Coding",
"Master and Topology",
"Admin UI and Workers",
"Shell",
"Security",
"Message Queue",
"Kubernetes, Helm and Deployment",
"Core and CLI",
"Documentation",
"Tests and CI",
"Other",
"Dependencies",
]
# Title scope -> section, first match wins. Matched against each lower-cased scope
# token, e.g. "filer.sync", "s3 lifecycle", "weed/s3api" (a "weed/" prefix is
# dropped first). Order matters: specific scopes come before the general ones
# they start with (s3tables before s3, volume.fsck before volume).
SCOPE_RULES = [
(r"(build\(deps\)|deps|dependabot)\b", "Dependencies"),
(r"(test|tests|e2e|integration)\b|.*\btests?$|test/", "Tests and CI"),
(r"(rust[ -]?volume|seaweed-volume|volume\(rust\)|rust)\b", "Rust Volume Server"),
(r"(s3 ?vectors?|vectors?|seaweed-vector|weed-vector|lance)\b", "Vectors"),
(r"(s3 ?tables|iceberg)\b", "S3 Tables and Iceberg"),
(r"(s3 sink|remote|remote_storage|filer|webdav|sftp|ftp|notification|replication|"
r"redis\d*|leveldb\d*|backup)\b", "Filer"),
(r"(s3|s3api|iam|oidc|sts|credential|sse|cors|policy|object ?lock)\b", "S3 API"),
(r"(mount|fuse)\b", "Mount"),
(r"(ec|ec_\w+|ec\.\w+|erasure[ _-]?coding|worker/ec)\b", "Erasure Coding"),
(r"(master|topology|wdclient|cluster|raft|placement)\b", "Master and Topology"),
(r"(admin|worker|weed-worker|plugin|plugin worker|maintenance)\b",
"Admin UI and Workers"),
(r"(shell|fsck|volume\.\w+|fs\.\w+|collection\.\w+|bucket\.\w+)", "Shell"),
(r"(volume|volume_\w+|storage|vacuum|operation|compression|needle)\b", "Volume Server"),
(r"(security|tls|mtls|jwt|grpc|kms)\b", "Security"),
(r"(mq|kafka|messaging)\b", "Message Queue"),
(r"(helm|k8s|kubernetes|operator|csi|docker|grafana|install\.sh|chart|snap|terraform)\b",
"Kubernetes, Helm and Deployment"),
(r"(docs?|readme|wiki)\b", "Documentation"),
(r"(ci|release|build|github|workflow|actions)\b", "Tests and CI"),
(r"(server|command|mini|weed-db|weed|util|pb|cmd|glog|stats|telemetry|query|config)\b",
"Core and CLI"),
]
# Changed path prefix -> section, for PRs whose title has no usable prefix.
# Longest prefix wins.
PATH_RULES = {
"weed/s3api/s3tables/": "S3 Tables and Iceberg",
"weed/s3api/iceberg/": "S3 Tables and Iceberg",
"weed/s3api/s3vectors/": "Vectors",
"weed/s3api/lance/": "Vectors",
"seaweed-vector/": "Vectors",
"weed/s3api/": "S3 API",
"weed/iam/": "S3 API",
"weed/iamapi/": "S3 API",
"weed/credential/": "S3 API",
"weed/filer/": "Filer",
"weed/filer_client/": "Filer",
"weed/server/filer": "Filer",
"weed/server/webdav": "Filer",
"weed/sftpd/": "Filer",
"weed/replication/": "Filer",
"weed/notification/": "Filer",
"weed/remote_storage/": "Filer",
"weed/mount/": "Mount",
"weed/storage/erasure_coding/": "Erasure Coding",
"weed/ec/": "Erasure Coding",
"weed/worker/tasks/erasure_coding/": "Erasure Coding",
"weed/storage/": "Volume Server",
"weed/server/volume": "Volume Server",
"weed/operation/": "Volume Server",
"seaweed-volume/": "Rust Volume Server",
"weed/topology/": "Master and Topology",
"weed/server/master": "Master and Topology",
"weed/wdclient/": "Master and Topology",
"weed/cluster/": "Master and Topology",
"weed/placement/": "Master and Topology",
"weed/admin/": "Admin UI and Workers",
"weed/worker/": "Admin UI and Workers",
"weed/plugin/": "Admin UI and Workers",
"seaweed-worker/": "Admin UI and Workers",
"weed/shell/": "Shell",
"weed/security/": "Security",
"weed/kms/": "Security",
"weed/mq/": "Message Queue",
"k8s/": "Kubernetes, Helm and Deployment",
"docker/": "Kubernetes, Helm and Deployment",
"snap/": "Kubernetes, Helm and Deployment",
"terraform/": "Kubernetes, Helm and Deployment",
"weed/": "Core and CLI",
"test/": "Tests and CI",
".github/": "Tests and CI",
}
PR_LINE = re.compile(r"^\* (?P<title>.+?) by @(?P<author>\S+) in https://github\.com/\S+/pull/(?P<num>\d+)\s*$")
# Lines regroup() writes between the PRs, so a second run accepts them.
SCAFFOLD = re.compile(r"^$|^### (%s)$|^<details><summary>Dependencies \(\d+ updates?\)</summary>$|^</details>$"
% "|".join(re.escape(s) for s in SECTIONS))
# "fix(s3tables,s3vectors): ...", "s3 lifecycle: ...", "[Volume] ...", "Fix/admin ..."
CONVENTIONAL = re.compile(r"^(?P<type>[a-z]+)\((?P<scope>[^)]*)\)!?$")
TYPE_ONLY = {"fix", "feat", "feature", "perf", "refactor", "chore", "style", "revert", "improvement"}
def scope_of(title):
"""Return the raw scope text of a PR title, or ""."""
t = title.strip()
m = re.match(r"^\[([^\]]{1,40})\]", t)
if m:
return m.group(1)
m = re.match(r"^(?:fix|feat|feature)/(\S+)", t, re.I)
if m:
return m.group(1)
head, sep, _ = t.partition(":")
if not sep or len(head) > 50:
return ""
head = head.strip()
m = CONVENTIONAL.match(head.lower())
if m:
kind, scope = m.group("type"), m.group("scope")
if kind in ("test", "tests", "ci"):
return "test"
if kind == "docs":
return "docs"
if kind == "build" and scope == "deps":
return "build(deps)"
return scope
if head.lower() in TYPE_ONLY:
return ""
return head
def section_by_scope(scope):
if scope.strip().lower() == "build(deps)":
return "Dependencies"
for token in re.split(r"\s*[,+&]\s*|\s+and\s+", scope.lower()):
token = token.strip()
token = re.sub(r"^weed[/ ]", "", token)
if not token:
continue
for pattern, section in SCOPE_RULES:
if re.match(pattern, token):
return section
return ""
def section_by_paths(paths):
votes = {}
code = [p for p in paths if not p.endswith(("_test.go", "_test.rs")) and not p.startswith("test/")]
for p in code or paths:
if p.endswith(".md"):
section = "Documentation"
else:
best = max((pre for pre in PATH_RULES if p.startswith(pre)), key=len, default="")
section = PATH_RULES.get(best, "Other")
votes[section] = votes.get(section, 0) + 1
if not votes:
return "Other"
# Most files wins; ties go to the earlier section in display order.
return max(votes, key=lambda s: (votes[s], -SECTIONS.index(s)))
def classify(title, author, pr_files):
if author.startswith("dependabot") or title.lower().startswith("build(deps)"):
return "Dependencies"
section = section_by_scope(scope_of(title))
if section:
return section
return section_by_paths(pr_files())
def split_notes(body):
"""Split generated notes into (PR lines, the text before them, the text after,
the non-blank lines of What's Changed)."""
lines = body.replace("\r\n", "\n").split("\n")
start = next((i for i, l in enumerate(lines) if l.strip() == "## What's Changed"), None)
if start is None:
return [], body, "", []
end = next((i for i in range(start + 1, len(lines))
if lines[i].startswith(("## ", "**Full Changelog**"))), len(lines))
prs, stray = [], []
block = [l.rstrip() for l in lines[start + 1:end] if l.strip()]
for line in lines[start + 1:end]:
if PR_LINE.match(line):
prs.append(line)
elif not SCAFFOLD.match(line.strip()):
stray.append(line)
if prs and stray:
raise ValueError("What's Changed has lines that are not PRs: %r" % stray[:3])
before = "\n".join(lines[:start]).rstrip()
after = "\n".join(lines[end:]).strip()
return prs, before, after, block
def regroup(body, pr_files):
prs, before, after, block = split_notes(body)
if not prs:
return body
groups = {s: [] for s in SECTIONS}
for line in prs:
m = PR_LINE.match(line)
num = int(m.group("num"))
groups[classify(m.group("title"), m.group("author"), lambda: pr_files(num))].append(line)
out = [before, ""] if before else []
out.append("## What's Changed")
for section in SECTIONS:
items = groups[section]
if not items:
continue
out.append("")
if section == "Dependencies":
out.append("<details><summary>Dependencies (%d update%s)</summary>" % (len(items), "" if len(items) == 1 else "s"))
out.append("")
out.extend(items)
out.append("")
out.append("</details>")
else:
out.append("### " + section)
out.extend(items)
if after:
out.extend(["", after])
notes = "\n".join(out).strip() + "\n"
# Headings are accepted only as this script's own grouping: a hand-written
# "### Security" over PRs of other components would otherwise be dropped.
if len(block) != len(prs) and split_notes(notes)[3] != block:
raise ValueError("What's Changed has headings that do not match the component grouping")
return notes
def gh(*args, stdin=None):
return subprocess.run(["gh", *args], input=stdin, capture_output=True, text=True, check=True).stdout
def pr_files(repo, num):
return gh("pr", "view", str(num), "-R", repo, "--json", "files", "--jq", ".files[].path").splitlines()
def main():
ap = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
ap.add_argument("--repo", default="seaweedfs/seaweedfs")
ap.add_argument("--tag", required=True)
ap.add_argument("--write", action="store_true", help="update the release notes on GitHub")
args = ap.parse_args()
body = json.loads(gh("release", "view", args.tag, "-R", args.repo, "--json", "body"))["body"]
failed = []
def files_of(num):
try:
return pr_files(args.repo, num)
except subprocess.CalledProcessError as e:
print("warning: files of #%d: %s" % (num, e.stderr.strip()), file=sys.stderr)
failed.append(num)
return []
try:
notes = regroup(body, files_of)
except ValueError as e:
sys.exit("release %s: %s" % (args.tag, e))
sys.stdout.write(notes)
if notes.strip() == body.replace("\r\n", "\n").strip():
print("release %s: notes already grouped, nothing to change" % args.tag, file=sys.stderr)
return
if failed and args.write:
sys.exit("release %s: not writing, could not read the files of %s"
% (args.tag, ", ".join("#%d" % n for n in failed)))
if args.write:
gh("release", "edit", args.tag, "-R", args.repo, "--notes-file", "-", stdin=notes)
print("release %s: notes updated" % args.tag, file=sys.stderr)
if __name__ == "__main__":
main()
@@ -1,190 +0,0 @@
import importlib.util
from pathlib import Path
import unittest
from unittest import mock
def load():
path = Path(__file__).with_name("release_notes_by_component.py")
spec = importlib.util.spec_from_file_location("release_notes_by_component", path)
module = importlib.util.module_from_spec(spec)
spec.loader.exec_module(module)
return module
rn = load()
def no_files(num):
raise AssertionError("files of #%d should not be needed" % num)
class ClassifyTest(unittest.TestCase):
def check(self, title, section, author="someone", files=None):
got = rn.classify(title, author, (lambda: files) if files is not None else (lambda: no_files(0)))
self.assertEqual(got, section, title)
def test_title_prefixes(self):
cases = {
"s3: keep serving for -preStopSeconds after SIGTERM": "S3 API",
"s3api: keep a multipart upload's state on the object's owner filer": "S3 API",
"fix(s3api): reject an invalid storage class before uploading the body": "S3 API",
"s3 lifecycle: count filer entries a walk reads but skips as progress": "S3 API",
"iam/oidc: accept a list audience": "S3 API",
"fix(s3tables,s3vectors): bind DeleteNamespace to the authorized records": "S3 Tables and Iceberg",
"iceberg maintenance: expire snapshots": "S3 Tables and Iceberg",
"s3vectors: support TagResource": "Vectors",
"s3 vectors: page ListVectors": "Vectors",
"vector: compact segments": "Vectors",
"filer: fail health check when absent from lock ring": "Filer",
"filer.sync: skip a rewritten source version": "Filer",
"filer.remote.sync: retry": "Filer",
"webdav: honor locks": "Filer",
"remote.uncache, remote.meta.sync, remote.cache: keep local writes": "Filer",
"mount: cache dir": "Mount",
"fix(mount): flush on close": "Mount",
"volume: keep reads over the download limit local": "Volume Server",
"[Volume] Preserve request semantics in replica proxy reads": "Volume Server",
"operation: pad the marginal gzip test": "Volume Server",
"fix(storage): update deletion metrics": "Volume Server",
"rust volume: bulk-load needle maps": "Rust Volume Server",
"seaweed-volume: pin the status": "Rust Volume Server",
"volume(rust): fix": "Rust Volume Server",
"ec: mount shards through the encode's .vif": "Erasure Coding",
"ec.rebuild: pick the right disk": "Erasure Coding",
"ec_balance: spread shards": "Erasure Coding",
"master: unregister a volume server whose heartbeats stop": "Master and Topology",
"topology: never report negative free volume slots": "Master and Topology",
"wdclient: let master-pushed adds replace locations": "Master and Topology",
"admin: manage filer webhook targets": "Admin UI and Workers",
"Fix/admin cluster submenu": "Admin UI and Workers",
"weed-worker: retry": "Admin UI and Workers",
"shell: fix s3.circuitBreaker help examples": "Shell",
"volume.fsck: skip deleted": "Shell",
"fix(fsck): inspect references to deleted volumes": "Shell",
"security: optional dialed-host check": "Security",
"helm: drop the 4.53 vector metrics render guard": "Kubernetes, Helm and Deployment",
"install.sh: add --enterprise": "Kubernetes, Helm and Deployment",
"mq/kafka: commit offsets": "Message Queue",
"server: add -grpc.localSocketDir": "Core and CLI",
"pb: recover when a local gRPC socket file is deleted": "Core and CLI",
"docs: explain the 3-filer recommendation": "Documentation",
"test(fuse): two mounts staging the same files": "Tests and CI",
"test/s3vectors: a hardened cluster": "Tests and CI",
"s3vectors test: cover tags": "Tests and CI",
"ci: run the Python 3.9 jobs on 3.10": "Tests and CI",
"build(deps): bump actions/setup-go from 6 to 7": "Dependencies",
}
for title, section in cases.items():
self.check(title, section)
def test_dependabot_author(self):
self.check("Bump x from 1 to 2", "Dependencies", author="dependabot[bot]")
def test_no_prefix_uses_files(self):
self.check("Fix s3 cancelled read relookup", "S3 API",
files=["weed/s3api/s3api_object_handlers.go", "weed/s3api/s3api_object_handlers_test.go"])
self.check("Retry transient failures in star history generation", "Tests and CI",
files=[".github/scripts/star_history.py"])
self.check("fix: handle empty chunk list", "Filer",
files=["weed/filer/filechunks.go", "weed/storage/needle/needle.go", "weed/filer/reader_at.go"])
self.check("Improve everything", "Other", files=[])
def test_test_files_count_only_when_alone(self):
self.check("Make it faster", "Mount",
files=["weed/mount/weedfs.go", "test/fuse/a_test.go", "test/fuse/b_test.go"])
self.check("Make it right", "S3 API",
files=["weed/s3api/auth.go", "test/s3/iam/test_a.py", "test/s3/iam/test_b.py"])
self.check("Cover more", "Tests and CI", files=["test/s3/iam/test_a.py"])
def test_path_with_space(self):
with mock.patch.object(rn, "gh", return_value="test/s3/iam/policy cases.json\n"):
files = rn.pr_files("seaweedfs/seaweedfs", 1)
self.assertEqual(files, ["test/s3/iam/policy cases.json"])
self.check("Cover more", "Tests and CI", files=files)
NOTES = """## What's Changed
* filer: fail health check when absent from lock ring by @a in https://github.com/seaweedfs/seaweedfs/pull/12709
* s3: answer ClientDisconnected by @b in https://github.com/seaweedfs/seaweedfs/pull/12707
* build(deps): bump actions/setup-go from 6 to 7 by @dependabot[bot] in https://github.com/seaweedfs/seaweedfs/pull/12685
* filer.sync: skip a rewritten source version by @a in https://github.com/seaweedfs/seaweedfs/pull/12366
* Fix s3 cancelled read relookup by @c in https://github.com/seaweedfs/seaweedfs/pull/12600
## New Contributors
* @c made their first contribution in https://github.com/seaweedfs/seaweedfs/pull/12600
**Full Changelog**: https://github.com/seaweedfs/seaweedfs/compare/4.53...4.54
"""
class RegroupTest(unittest.TestCase):
def files(self, num):
self.assertEqual(num, 12600)
return ["weed/s3api/s3api_object_handlers.go"]
def test_regroup(self):
out = rn.regroup(NOTES, self.files)
self.assertEqual(out, """## What's Changed
### S3 API
* s3: answer ClientDisconnected by @b in https://github.com/seaweedfs/seaweedfs/pull/12707
* Fix s3 cancelled read relookup by @c in https://github.com/seaweedfs/seaweedfs/pull/12600
### Filer
* filer: fail health check when absent from lock ring by @a in https://github.com/seaweedfs/seaweedfs/pull/12709
* filer.sync: skip a rewritten source version by @a in https://github.com/seaweedfs/seaweedfs/pull/12366
<details><summary>Dependencies (1 update)</summary>
* build(deps): bump actions/setup-go from 6 to 7 by @dependabot[bot] in https://github.com/seaweedfs/seaweedfs/pull/12685
</details>
## New Contributors
* @c made their first contribution in https://github.com/seaweedfs/seaweedfs/pull/12600
**Full Changelog**: https://github.com/seaweedfs/seaweedfs/compare/4.53...4.54
""")
def test_idempotent(self):
once = rn.regroup(NOTES, self.files)
self.assertEqual(rn.regroup(once, self.files), once)
def test_every_pr_kept(self):
out = rn.regroup(NOTES, self.files)
for n in (12709, 12707, 12685, 12366, 12600):
self.assertEqual(out.count("/pull/%d\n" % n), 2 if n == 12600 else 1)
def test_changelog_kept_without_new_contributors(self):
body = NOTES.replace(
"## New Contributors\n* @c made their first contribution in https://github.com/seaweedfs/seaweedfs/pull/12600\n\n", "")
out = rn.regroup(body, self.files)
self.assertTrue(out.endswith("</details>\n\n**Full Changelog**: https://github.com/seaweedfs/seaweedfs/compare/4.53...4.54\n"), out)
self.assertEqual(rn.regroup(out, self.files), out)
def test_unknown_line_refused(self):
body = NOTES.replace("## New Contributors", "* a hand-written note\n\n## New Contributors")
with self.assertRaises(ValueError):
rn.regroup(body, self.files)
with self.assertRaises(ValueError):
rn.regroup(NOTES.replace("## What's Changed\n", "## What's Changed\n### Known issue\n"), self.files)
def test_hand_written_component_heading_refused(self):
with self.assertRaises(ValueError):
rn.regroup(NOTES.replace("## What's Changed\n", "## What's Changed\n### Security\n"), self.files)
once = rn.regroup(NOTES, self.files)
with self.assertRaises(ValueError):
rn.regroup(once.replace("\n### Filer\n", "\n ### Filer\n"), self.files)
def test_author_from_the_by_clause(self):
body = "## What's Changed\n* Bump a by @dependabot[bot] in docs by @alice in https://github.com/seaweedfs/seaweedfs/pull/7\n"
self.assertIn("### Documentation", rn.regroup(body, lambda n: ["README.md"]))
def test_notes_without_prs_unchanged(self):
body = "Hand-written notes\n\n**Full Changelog**: x"
self.assertEqual(rn.regroup(body, no_files), body)
if __name__ == "__main__":
unittest.main()
-11
View File
@@ -5,13 +5,11 @@ on:
branches: [ master ]
paths:
- '.github/actions/**'
- '.github/scripts/*release_notes_by_component.py'
- '.github/workflows/action-tests.yml'
pull_request:
branches: [ master ]
paths:
- '.github/actions/**'
- '.github/scripts/*release_notes_by_component.py'
- '.github/workflows/action-tests.yml'
concurrency:
@@ -30,12 +28,3 @@ jobs:
- name: Run cleanup action unit tests
run: python3 .github/actions/cleanup-stale-fuse-mounts/test_cleanup.py
release-notes-by-component:
runs-on: ubuntu-latest
steps:
- name: Check out code
uses: actions/checkout@v7
- name: Run release notes grouping unit tests
run: python3 .github/scripts/test_release_notes_by_component.py
+54 -20
View File
@@ -259,6 +259,40 @@ jobs:
grep -q "security-config" /tmp/security.yaml
echo "Security configuration renders correctly"
echo "=== Testing volume server HTTPS ==="
awk '/^ *\[https.client\]$/{f=1;next} /^ *\[/{f=0} f' /tmp/security.yaml | grep -q 'enabled = false'
helm template test $CHART_DIR \
--set global.seaweedfs.enableSecurity=true \
--set global.seaweedfs.volumeHttps.enabled=true \
--set volumes.hot.replicas=1 \
> /tmp/volume-https.yaml
awk '/^ *\[https.client\]$/{f=1;next} /^ *\[/{f=0} f' /tmp/volume-https.yaml > /tmp/volume-https-client.txt
grep -q 'enabled = true' /tmp/volume-https-client.txt
grep -q 'cert = "/usr/local/share/ca-certificates/client/tls.crt"' /tmp/volume-https-client.txt
grep -q 'ca = "/usr/local/share/ca-certificates/ca/tls.crt"' /tmp/volume-https-client.txt
awk '/^ *\[https.volume\]$/{f=1;next} /^ *\[/{f=0} f' /tmp/volume-https.yaml \
| grep -q 'cert = "/usr/local/share/ca-certificates/volume/tls.crt"'
grep -q "'\*.test-seaweedfs-volume.default'" /tmp/volume-https.yaml
grep -q "'\*.test-seaweedfs-volume-hot.default'" /tmp/volume-https.yaml
test "$(grep -c 'scheme: HTTPS' /tmp/volume-https.yaml)" -eq 4
test "$(grep -c 'checksum/volume-https:' /tmp/volume-https.yaml)" -eq 4
if grep -q 'checksum/volume-https' /tmp/security.yaml; then
echo "FAIL: the volume HTTPS rollout annotation should only render while the option is on"
exit 1
fi
if helm template test $CHART_DIR --set global.seaweedfs.volumeHttps.enabled=true > /dev/null 2>/tmp/volume-https-nosec.err; then
echo "FAIL: volumeHttps without enableSecurity should fail to render"
exit 1
fi
grep -q "requires global.seaweedfs.enableSecurity" /tmp/volume-https-nosec.err
if helm template test $CHART_DIR --set global.seaweedfs.enableSecurity=true \
--set global.seaweedfs.volumeHttps.enabled=true --set allInOne.enabled=true > /dev/null 2>/tmp/volume-https-aio.err; then
echo "FAIL: volumeHttps with allInOne should fail to render"
exit 1
fi
grep -q "not supported with allInOne" /tmp/volume-https-aio.err
echo "Volume server HTTPS renders [https.volume], [https.client], certificate names, HTTPS probes and rollout annotations"
echo "=== Testing secure bucket-creation hook certificate mounts ==="
helm template test $CHART_DIR \
--show-only templates/shared/post-install-bucket-hook.yaml \
@@ -360,9 +394,9 @@ jobs:
def docs(manifest):
return [d for d in yaml.safe_load_all(manifest) if d]
def configmap(manifest, name):
def security_config(manifest, name):
for d in docs(manifest):
if d.get("kind") == "ConfigMap" and d["metadata"]["name"] == name:
if d.get("kind") == "Secret" and d["metadata"]["name"] == name:
return d
return None
@@ -387,13 +421,13 @@ jobs:
# security-related; preserve that so this PR is non-breaking on
# existing installs.
out = render({})
if configmap(out, "test-seaweedfs-security-config") is not None:
failed.append("defaults: security ConfigMap should not render")
if security_config(out, "test-seaweedfs-security-config") is not None:
failed.append("defaults: security Secret should not render")
else:
print("defaults: no security-config ConfigMap (unchanged)")
print("defaults: no security-config Secret (unchanged)")
# Case 2: filerWrite=true alone is the documented opt-in for
# the Admin UI Users tab. Configmap must render with
# the Admin UI Users tab. The Secret must render with
# [jwt.filer_signing] and NO [grpc.*] sections (cert paths
# only exist with mTLS).
out = render({
@@ -403,11 +437,11 @@ jobs:
# refuses to bind without authentication.
"admin.secret.adminPassword": "ci-admin-password",
})
cm = configmap(out, "test-seaweedfs-security-config")
cm = security_config(out, "test-seaweedfs-security-config")
if cm is None:
failed.append("filerWrite=true: security ConfigMap missing")
failed.append("filerWrite=true: security Secret missing")
else:
toml = cm["data"]["security.toml"]
toml = cm["stringData"]["security.toml"]
if "[jwt.filer_signing]" not in toml:
failed.append("filerWrite=true: security.toml missing [jwt.filer_signing]")
if "[grpc" in toml:
@@ -415,7 +449,7 @@ jobs:
if "[jwt.filer_signing]" in toml and "[grpc" not in toml:
print("filerWrite=true: security.toml has [jwt.filer_signing], no [grpc.*]")
# Case 3: filer + admin pods must MOUNT the security ConfigMap
# Case 3: filer + admin pods must MOUNT the security Secret
# under filerWrite=true so the JWT key reaches both processes.
# Cert volumes must NOT be present (no mTLS).
for wl in ("test-seaweedfs-filer", "test-seaweedfs-admin"):
@@ -435,11 +469,11 @@ jobs:
# with both [jwt.signing] and [grpc.*]. Guards against the
# decoupling change accidentally regressing the mTLS path.
out = render({"global.seaweedfs.enableSecurity": "true"})
cm = configmap(out, "test-seaweedfs-security-config")
cm = security_config(out, "test-seaweedfs-security-config")
if cm is None:
failed.append("enableSecurity=true: security ConfigMap missing")
failed.append("enableSecurity=true: security Secret missing")
else:
toml = cm["data"]["security.toml"]
toml = cm["stringData"]["security.toml"]
missing = [s for s in ("[jwt.signing]", "[grpc.master]") if s not in toml]
if missing:
failed.append(f"enableSecurity=true: security.toml missing {missing}")
@@ -456,8 +490,8 @@ jobs:
except subprocess.CalledProcessError as e:
failed.append(f"{null_path}=null: render failed: {e.output[:200] if e.output else e}")
continue
if configmap(out, "test-seaweedfs-security-config") is not None:
failed.append(f"{null_path}=null: should not render configmap")
if security_config(out, "test-seaweedfs-security-config") is not None:
failed.append(f"{null_path}=null: should not render the security Secret")
else:
print(f"{null_path}=null: render tolerates explicit null")
@@ -561,16 +595,16 @@ jobs:
mount = next((m for m in pod["containers"][0].get("volumeMounts", []) if m["name"] == "security-config"), None)
# The file has to land on one of weed's config search paths, so
# the target matters as much as the volume itself - and a volume
# of that name backed by anything but the chart's ConfigMap
# of that name backed by anything but the chart's Secret
# leaves `weed shell` without the signing key just the same.
target = ("/etc/seaweedfs/security.toml", "security.toml")
configmap = "test-seaweedfs-security-config"
secret = "test-seaweedfs-security-config"
if vol is None or mount is None:
failed.append("filerWrite=true: bucket hook does not mount security-config (s3.configure would fail)")
elif (mount.get("mountPath"), mount.get("subPath")) != target:
failed.append(f"filerWrite=true: bucket hook mounts security-config as {mount}")
elif vol.get("configMap", {}).get("name") != configmap:
failed.append(f"filerWrite=true: security-config is not backed by the {configmap} ConfigMap: {vol}")
elif vol.get("secret", {}).get("secretName") != secret:
failed.append(f"filerWrite=true: security-config is not backed by the {secret} Secret: {vol}")
else:
print("filerWrite=true: bucket hook mounts security-config at /etc/seaweedfs/security.toml")
@@ -2076,7 +2110,7 @@ jobs:
"--set", "admin.enabled=true", "--set", "admin.secret.adminPassword=ci",
"--set", "cosi.enabled=true", "--set", "networkPolicy.enabled=true",
"--set", "volumes.dc2.replicas=1",
# render the security ConfigMap without its generated jwt key
# render the security Secret without its generated jwt key
"--set", "global.seaweedfs.enableSecurity=true",
"--set", "global.seaweedfs.securityConfig.jwtSigning.volumeWrite=false",
]
+1 -17
View File
@@ -4,9 +4,7 @@ name: "release: bump version and cut the release"
# 1. bump MAJOR/MINOR in constants.go and the Helm Chart.yaml, commit to master
# 2. push the <appVersion> tag, which fans out to the workflows that trigger on
# `push: tags` (binaries_release*, container_release_unified, helm_manual_release)
# 3. create the GitHub release, with GitHub's generated notes, then regroup those
# notes by component (S3 API, Filer, Mount, ...) with
# .github/scripts/release_notes_by_component.py
# 3. create the GitHub release, with GitHub's generated notes
# 4. dispatch "Prepare release" in seaweedfs-csi-driver and seaweedfs-operator,
# which pick up the new master through `go get -u`, and wait for both
#
@@ -43,7 +41,6 @@ jobs:
runs-on: ubuntu-latest
permissions:
contents: write
pull-requests: read # the notes step reads a PR's files when its title has no component prefix
outputs:
app_version: ${{ steps.compute.outputs.app_version }}
sha: ${{ steps.tag.outputs.sha }}
@@ -177,19 +174,6 @@ jobs:
TAG: ${{ steps.compute.outputs.app_version }}
run: gh release create "$TAG" --title "$TAG" --generate-notes --verify-tag
# GitHub's generated notes are one flat list of PRs. Sort them under
# component headings by title prefix (or changed files). The release is
# already published, so a failure here only leaves the flat list; rerun
# the script by hand to fix it:
# python3 .github/scripts/release_notes_by_component.py --tag <tag> --write
- name: Group the release notes by component
if: ${{ !inputs.dry_run }}
continue-on-error: true
env:
GH_TOKEN: ${{ secrets.RELEASE_PAT || secrets.GITHUB_TOKEN }}
TAG: ${{ steps.compute.outputs.app_version }}
run: python3 .github/scripts/release_notes_by_component.py --repo "$GITHUB_REPOSITORY" --tag "$TAG" --write > /dev/null
downstream:
needs: release
if: ${{ inputs.downstream && !inputs.dry_run }}
@@ -0,0 +1,74 @@
name: "S3 listing study tests"
on:
push:
branches: [ master ]
paths:
- 'weed/s3api/**'
- 'weed/filer/**'
- 'weed/server/**'
- 'test/s3/listing_study/**'
- '.github/workflows/s3-listing-study-tests.yml'
pull_request:
branches: [ master ]
paths:
- 'weed/s3api/**'
- 'weed/filer/**'
- 'weed/server/**'
- 'test/s3/listing_study/**'
- '.github/workflows/s3-listing-study-tests.yml'
concurrency:
group: ${{ github.event.pull_request.number || github.ref }}/s3-listing-study-tests
cancel-in-progress: true
permissions:
contents: read
jobs:
listing-study:
name: s3-listing-study tools and strategies
runs-on: ubuntu-24.04
timeout-minutes: 40
defaults:
run:
working-directory: test/s3/listing_study
steps:
- name: Check out code
uses: actions/checkout@v7
with:
persist-credentials: false
- name: Set up Go
uses: actions/setup-go@v7
with:
go-version-file: 'go.mod'
cache: ${{ runner.environment == 'github-hosted' }}
- name: Cache pinned listing tools
uses: actions/cache@v6
with:
path: test/s3/listing_study/.work/tools
key: listing-study-tools-${{ runner.os }}-${{ hashFiles('test/s3/listing_study/Makefile', 'test/s3/listing_study/install_tools.py') }}
- name: Build weed and install the study's pinned tools
run: make build-weed tools
- name: Run the listing study against SeaweedFS
timeout-minutes: 15
run: make test
- name: Show logs on failure
if: failure()
run: |
tail -200 .work/weed.log || true
- name: Upload results
if: always()
uses: actions/upload-artifact@v7
with:
name: s3-listing-study-results
path: |
test/s3/listing_study/.work/run/results.json
test/s3/listing_study/.work/weed.log
retention-days: 14
+11 -1
View File
@@ -68,7 +68,15 @@ jobs:
with:
java-version: '17'
distribution: 'temurin'
cache: ${{ runner.environment == 'github-hosted' && 'maven' || '' }}
- name: Cache suite Maven dependencies
if: runner.environment == 'github-hosted'
uses: actions/cache@v6
with:
path: ~/.m2/repository
key: snowflake-s3compat-m2-${{ hashFiles('test/s3/snowflake/run.sh') }}
restore-keys: |
snowflake-s3compat-m2-
- name: Install SeaweedFS
run: |
@@ -78,6 +86,8 @@ jobs:
- name: Run Snowflake S3Compat API tests
timeout-minutes: 20
env:
MAVEN_ARGS: -ntp -s ${{ github.workspace }}/test/s3/snowflake/maven-settings.xml
run: |
# Starts weed server, creates the buckets/objects the suite needs,
# clones the upstream suite, and runs mvn -Dtest=S3CompatApiTest.
+6
View File
@@ -684,6 +684,12 @@ jobs:
pull() { for i in 1 2 3; do docker pull "$1" && return 0; sleep 15; done; return 1; }
pull apache/spark:3.5.1
- name: Cache Iceberg jars
uses: actions/cache@v6
with:
path: ~/.cache/seaweedfs-test-jars
key: iceberg-jars-${{ hashFiles('test/s3tables/catalog_spark/setup_test.go') }}
- name: Run go mod tidy
run: go mod tidy
@@ -0,0 +1,57 @@
name: "S3 Tigris Compatibility Tests"
on:
push:
branches: [ master ]
paths:
- 'weed/s3api/**'
- 'weed/server/filer_server_handlers_*.go'
- 'test/s3/tigris_compat/**'
- '.github/workflows/s3-tigris-compat-tests.yml'
pull_request:
branches: [ master ]
paths:
- 'weed/s3api/**'
- 'weed/server/filer_server_handlers_*.go'
- 'test/s3/tigris_compat/**'
- '.github/workflows/s3-tigris-compat-tests.yml'
concurrency:
group: ${{ github.workflow }}-${{ github.event.pull_request.number || (github.event_name == 'push' && github.ref) || github.run_id }}
cancel-in-progress: ${{ github.event_name == 'pull_request' || (github.event_name == 'push' && !startsWith(github.ref, 'refs/tags/')) }}
permissions:
contents: read
jobs:
s3-tigris-compat-tests:
name: s3-api-compat-tests (Tigris)
runs-on: ubuntu-24.04
timeout-minutes: 20
steps:
- name: Check out code
uses: actions/checkout@v7
- name: Set up Go
uses: actions/setup-go@v7
with:
go-version-file: 'go.mod'
cache: ${{ runner.environment == 'github-hosted' }}
- name: Run s3-api-compat-tests
working-directory: test/s3/tigris_compat
run: make test
- name: Show server log on failure
if: failure()
working-directory: test/s3/tigris_compat
run: tail -200 weed-test.log || true
- name: Upload report
if: always()
uses: actions/upload-artifact@v7
with:
name: s3-tigris-compat-results
path: test/s3/tigris_compat/results.json
if-no-files-found: ignore
retention-days: 7
@@ -6,12 +6,16 @@ on:
paths:
- 'telemetry/**'
- 'weed/telemetry/**'
- 'go.mod'
- 'go.sum'
- '.github/workflows/telemetry-integration.yml'
pull_request:
branches: [ master ]
paths:
- 'telemetry/**'
- 'weed/telemetry/**'
- 'go.mod'
- 'go.sum'
- '.github/workflows/telemetry-integration.yml'
concurrency:
+19 -4
View File
@@ -30,10 +30,12 @@ so your deployment will be spread/HA.
The chart's templates render with Helm 3.16.3 and newer versions tested in CI.
Earlier Helm versions are not covered by this chart's compatibility checks.
When JWT signing is enabled, a live upgrade reuses keys from the existing
security ConfigMap (including the legacy ConfigMap name); a key absent from
that ConfigMap is generated. Plain `helm template` does not read cluster state;
use an install and upgrade against a cluster to check key persistence.
When JWT signing is enabled, `security.toml` is kept in the
`<fullname>-security-config` Secret. A live upgrade reuses keys from that
Secret, or from the security ConfigMap of earlier chart versions (including
the legacy ConfigMap name); a key absent from them is generated. Plain
`helm template` does not read cluster state; use an install and upgrade
against a cluster to check key persistence.
The key reader supports single-line quoted TOML strings in JWT sections,
including indented assignments, quoted key names, and bare or simply quoted
@@ -569,6 +571,8 @@ s3:
With `rules` left empty the route sends all traffic to the component's own Service and port. A rule may set `matches`, `filters`, `timeouts` and `backendRefs`, and a rule without `backendRefs` still routes to that Service. In all-in-one mode the master, volume, filer and s3 routes target the all-in-one Service. When S3 runs only on the filer (`filer.s3.enabled` without `s3.enabled`), the s3 route uses `filer.s3.port`.
Outside all-in-one mode, the volume Ingress and the default volume route target the `client-volume` Service, which lists only Ready volume pods. The `volume` Service also publishes pods that are not Ready yet, so their DNS records can exist before readiness. An explicit `backendRefs` entry that names the `volume` Service can therefore reach a volume server before it is Ready. Name the `client-volume` Service instead, with `volume.port`, and copy its `metadata.name` from `helm template` because long release names are truncated.
## Network Policies
In a namespace with a default-deny policy the install hangs: the components cannot resolve each other, and the post-install bucket hook waits on the master and filer until it gives up. `networkPolicy.enabled` renders one `NetworkPolicy` per component, selecting its pods by the standard `app.kubernetes.io/{name,instance,component}` labels and admitting traffic from the other pods of the release on the ports that component listens on.
@@ -610,6 +614,17 @@ Two things worth knowing before you turn this on:
The DNS selectors default to CoreDNS as kubeadm, kind and the managed offerings from AWS, Google and Azure install it. On OpenShift, override `egress.dnsNamespaceSelector` and `egress.dnsPodSelector` to match `openshift-dns`; see the comment in `values.yaml`.
## Volume server HTTPS
With `global.seaweedfs.enableSecurity`, gRPC between components uses mutual TLS, but the volume servers' HTTP port, which carries the file data, stays plaintext. Set `global.seaweedfs.volumeHttps.enabled: true` to serve it over TLS:
- `[https.volume]` points at the volume certificate, which then also names every volume group's service (`*.<fullname>-volume.<namespace>`), since volume servers register as `<pod>.<fullname>-volume.<namespace>`. With `certificates.externalCertificates`, the provided `<fullname>-volume-cert` needs those names.
- `[https.client]` is enabled with the client certificate and the chart CA, so every chart pod that talks to the volume servers does it over HTTPS.
- The volume probes use HTTPS.
- Every pod that reads `security.toml` gets a `checksum/volume-https` annotation, so switching the option on or off restarts them together.
The volume servers do not require client certificates, because kubelet probes cannot present one; enable JWT signing (`securityConfig.jwtSigning`) to authorize volume reads and writes. `[https.client]` also sends the admin UI's HTTP lookups on masters and S3 over HTTPS, which their plain listeners refuse, so the UI can miss volume public URLs and refuse SSE-S3 vector buckets. A volume Ingress or HTTPRoute needs its controller's HTTPS-backend setting. `weed mount`, the CSI driver and other clients outside the chart need their own `[https.client]` with the chart CA. The option is not supported with `allInOne`, whose volume server registers by pod IP.
## Common labels
`commonLabels` adds labels to the metadata of every object the chart creates. It may be a map or a template string rendering to one. The chart's own labels cannot be overridden, and pods (see `podLabels`) and `volumeClaimTemplates` are left unchanged.
@@ -56,6 +56,7 @@ spec:
{{- with .Values.admin.podAnnotations }}
{{- toYaml . | nindent 8 }}
{{- end }}
{{- with include "seaweedfs.volumeHttpsAnnotation" $ }}{{ . | nindent 8 }}{{ end }}
spec:
restartPolicy: {{ default .Values.global.seaweedfs.restartPolicy .Values.admin.restartPolicy }}
{{- if .Values.admin.affinity }}
@@ -313,8 +314,8 @@ spec:
{{- end }}
{{- if include "seaweedfs.securityConfigEnabled" . }}
- name: security-config
configMap:
name: {{ include "seaweedfs.fullname" . }}-security-config
secret:
secretName: {{ include "seaweedfs.fullname" . }}-security-config
{{- end }}
{{- if .Values.global.seaweedfs.enableSecurity }}
- name: ca-cert
@@ -1,5 +1,8 @@
{{- include "seaweedfs.compat" . -}}
{{- if .Values.allInOne.enabled }}
{{- if include "seaweedfs.volumeHttps" . }}
{{- fail "global.seaweedfs.volumeHttps is not supported with allInOne: its volume server registers by pod IP, which the volume certificate cannot name" }}
{{- end }}
{{- $s3PreStopSeconds := 0 }}
{{- if .Values.allInOne.s3.enabled }}
{{- $s3PreStopSeconds = .Values.allInOne.s3.preStopSeconds }}
@@ -504,8 +507,8 @@ spec:
name: {{ include "seaweedfs.fullname" . }}-master-config
{{- if include "seaweedfs.securityConfigEnabled" . }}
- name: security-config
configMap:
name: {{ include "seaweedfs.fullname" . }}-security-config
secret:
secretName: {{ include "seaweedfs.fullname" . }}-security-config
{{- end }}
{{- if .Values.global.seaweedfs.enableSecurity }}
- name: ca-cert
@@ -38,6 +38,14 @@ spec:
- '*.{{ include "seaweedfs.fullname" . }}-master.{{ include "seaweedfs.namespace" . }}'
- '*.{{ include "seaweedfs.fullname" . }}-master.{{ include "seaweedfs.namespace" . }}.svc'
- '*.{{ include "seaweedfs.fullname" . }}-master.{{ include "seaweedfs.namespace" . }}.svc.cluster.local'
{{- if include "seaweedfs.volumeHttps" . }}
{{- range $vname, $_ := deepCopy .Values.volumes | mergeOverwrite (dict "" .Values.volume) }}
{{- $volumeService := printf "%s.%s" (include "seaweedfs.componentName" (list $ (trimSuffix "-" (printf "volume-%s" $vname)))) (include "seaweedfs.namespace" $) }}
- '*.{{ $volumeService }}'
- '*.{{ $volumeService }}.svc'
- '*.{{ $volumeService }}.svc.cluster.local'
{{- end }}
{{- end }}
{{- if .Values.certificates.ipAddresses }}
ipAddresses:
{{- range .Values.certificates.ipAddresses }}
@@ -39,6 +39,7 @@ spec:
{{- with .Values.cosi.podAnnotations }}
{{- toYaml . | nindent 8 }}
{{- end }}
{{- with include "seaweedfs.volumeHttpsAnnotation" $ }}{{ . | nindent 8 }}{{ end }}
spec:
restartPolicy: {{ default .Values.global.seaweedfs.restartPolicy .Values.cosi.restartPolicy }}
{{- if .Values.cosi.affinity }}
@@ -195,8 +196,8 @@ spec:
{{- end }}
{{- if include "seaweedfs.securityConfigEnabled" . }}
- name: security-config
configMap:
name: {{ include "seaweedfs.fullname" . }}-security-config
secret:
secretName: {{ include "seaweedfs.fullname" . }}-security-config
{{- end }}
{{- if .Values.global.seaweedfs.enableSecurity }}
- name: ca-cert
@@ -57,6 +57,7 @@ spec:
{{- end }}
annotations:
{{- toYaml $podAnnotations | nindent 8 }}
{{- with include "seaweedfs.volumeHttpsAnnotation" $ }}{{ . | nindent 8 }}{{ end }}
spec:
restartPolicy: {{ default .Values.global.seaweedfs.restartPolicy .Values.filer.restartPolicy }}
{{- if .Values.filer.affinity }}
@@ -452,8 +453,8 @@ spec:
{{- end }}
{{- if include "seaweedfs.securityConfigEnabled" . }}
- name: security-config
configMap:
name: {{ include "seaweedfs.fullname" . }}-security-config
secret:
secretName: {{ include "seaweedfs.fullname" . }}-security-config
{{- end }}
{{- if .Values.global.seaweedfs.enableSecurity }}
- name: ca-cert
@@ -48,6 +48,7 @@ spec:
{{- $_ := set $podAnnotations "checksum/master-config" (include "seaweedfs.master.configChecksum" .) }}
annotations:
{{- toYaml $podAnnotations | nindent 8 }}
{{- with include "seaweedfs.volumeHttpsAnnotation" $ }}{{ . | nindent 8 }}{{ end }}
spec:
restartPolicy: {{ default .Values.global.seaweedfs.restartPolicy .Values.master.restartPolicy }}
{{- if .Values.master.affinity }}
@@ -299,8 +300,8 @@ spec:
name: {{ include "seaweedfs.fullname" . }}-master-config
{{- if include "seaweedfs.securityConfigEnabled" . }}
- name: security-config
configMap:
name: {{ include "seaweedfs.fullname" . }}-security-config
secret:
secretName: {{ include "seaweedfs.fullname" . }}-security-config
{{- end }}
{{- if .Values.global.seaweedfs.enableSecurity }}
- name: ca-cert
@@ -54,6 +54,7 @@ spec:
{{- end }}
annotations:
{{- toYaml $podAnnotations | nindent 8 }}
{{- with include "seaweedfs.volumeHttpsAnnotation" $ }}{{ . | nindent 8 }}{{ end }}
spec:
restartPolicy: {{ default .Values.global.seaweedfs.restartPolicy .Values.s3.restartPolicy }}
{{- if .Values.s3.affinity }}
@@ -288,8 +289,8 @@ spec:
{{- end }}
{{- if include "seaweedfs.securityConfigEnabled" . }}
- name: security-config
configMap:
name: {{ include "seaweedfs.fullname" . }}-security-config
secret:
secretName: {{ include "seaweedfs.fullname" . }}-security-config
{{- end }}
{{- if .Values.global.seaweedfs.enableSecurity }}
- name: ca-cert
@@ -43,6 +43,7 @@ spec:
{{- with .Values.sftp.podAnnotations }}
{{- toYaml . | nindent 8 }}
{{- end }}
{{- with include "seaweedfs.volumeHttpsAnnotation" $ }}{{ . | nindent 8 }}{{ end }}
spec:
restartPolicy: {{ default .Values.global.seaweedfs.restartPolicy .Values.sftp.restartPolicy }}
{{- if .Values.sftp.affinity }}
@@ -297,8 +298,8 @@ spec:
{{- end }}
{{- if include "seaweedfs.securityConfigEnabled" . }}
- name: security-config
configMap:
name: {{ include "seaweedfs.fullname" . }}-security-config
secret:
secretName: {{ include "seaweedfs.fullname" . }}-security-config
{{- end }}
{{- if .Values.global.seaweedfs.enableSecurity }}
- name: ca-cert
@@ -491,6 +491,25 @@ true
{{- end -}}
{{- end -}}
{{/* True when the volume servers serve HTTPS. Their certificates come from
enableSecurity, so the option fails without it. */}}
{{- define "seaweedfs.volumeHttps" -}}
{{- if ((.Values.global.seaweedfs).volumeHttps).enabled -}}
{{- if not .Values.global.seaweedfs.enableSecurity -}}
{{- fail "global.seaweedfs.volumeHttps.enabled requires global.seaweedfs.enableSecurity, which provides the certificates" -}}
{{- end -}}
true
{{- end -}}
{{- end -}}
{{/* Pod annotation that restarts the readers of security.toml when volume
HTTPS is switched on or off. */}}
{{- define "seaweedfs.volumeHttpsAnnotation" -}}
{{- with include "seaweedfs.volumeHttps" . -}}
checksum/volume-https: {{ sha256sum . | quote }}
{{- end -}}
{{- end -}}
{{/* Read a JWT key from the chart's existing security.toml without fromToml
(which requires Helm >=3.17). Args: (list "<section>" $raw).
Return the single-line TOML string token, including its quotes, so escapes
@@ -286,8 +286,8 @@ spec:
{{- end }}
{{- if include "seaweedfs.securityConfigEnabled" . }}
- name: security-config
configMap:
name: {{ include "seaweedfs.fullname" . }}-security-config
secret:
secretName: {{ include "seaweedfs.fullname" . }}-security-config
{{- end }}
{{- if .Values.global.seaweedfs.enableSecurity }}
- name: ca-cert
@@ -1,7 +1,7 @@
{{- include "seaweedfs.compat" . -}}
{{- if include "seaweedfs.securityConfigEnabled" . }}
apiVersion: v1
kind: ConfigMap
kind: Secret
metadata:
name: {{ include "seaweedfs.fullname" . }}-security-config
namespace: {{ include "seaweedfs.namespace" . | quote }}
@@ -11,7 +11,8 @@ metadata:
app.kubernetes.io/managed-by: {{ .Release.Service }}
app.kubernetes.io/instance: {{ .Release.Name }}
{{- with include "seaweedfs.commonLabels" $ }}{{ . | nindent 4 }}{{ end }}
data:
type: Opaque
stringData:
{{- $fullname := include "seaweedfs.fullname" . }}
{{- $securityConfigName := printf "%s-security-config" $fullname }}
{{- $existing := lookup "v1" "ConfigMap" (include "seaweedfs.namespace" .) $securityConfigName }}
@@ -19,7 +20,11 @@ data:
{{- $legacyName := printf "%s-%s" (include "seaweedfs.name" .) "security-config" }}
{{- $existing = lookup "v1" "ConfigMap" (include "seaweedfs.namespace" .) $legacyName }}
{{- end }}
{{- $existingToml := dig "data" "security.toml" "" $existing }}
{{- $ownConfigMap := eq (dig "metadata" "annotations" "meta.helm.sh/release-name" "" $existing) .Release.Name }}
{{- if and $ownConfigMap (or (and .Values.master.enabled (gt (int .Values.master.updatePartition) 0)) (and .Values.filer.enabled (gt (int .Values.filer.updatePartition) 0))) }}
{{- fail "this upgrade moves security.toml from a ConfigMap to a Secret and removes the ConfigMap; run it with master.updatePartition=0 and filer.updatePartition=0 so no pod is left on the old template" }}
{{- end }}
{{- $existingToml := dig "data" "security.toml" "" (lookup "v1" "Secret" (include "seaweedfs.namespace" .) $securityConfigName) | b64dec | default (dig "data" "security.toml" "" $existing) }}
{{- $securityConfigValues := .Values.global.seaweedfs.securityConfig | default dict }}
{{- $jwtSigning := $securityConfigValues.jwtSigning | default dict }}
{{- $expiresAfterSeconds := $jwtSigning.expiresAfterSeconds | default dict }}
@@ -30,7 +35,7 @@ data:
# the jwt signing key is read by master and volume server
# the jwt defaults to expire after 10 seconds
[jwt.signing]
key = {{ include "seaweedfs.existingTomlKey" (list "jwt.signing" $existingToml) | default (randAlphaNum 10 | b64enc | quote) }}
key = {{ include "seaweedfs.existingTomlKey" (list "jwt.signing" $existingToml) | default (randAlphaNum 64 | quote) }}
{{- if gt (int $expiresAfterSeconds.volumeWrite) 0 }}
expires_after_seconds = {{ int $expiresAfterSeconds.volumeWrite }}
{{- end }}
@@ -42,7 +47,7 @@ data:
# - the Volume server validates the JWT on reading
# the jwt defaults to expire after 60 seconds
[jwt.signing.read]
key = {{ include "seaweedfs.existingTomlKey" (list "jwt.signing.read" $existingToml) | default (randAlphaNum 10 | b64enc | quote) }}
key = {{ include "seaweedfs.existingTomlKey" (list "jwt.signing.read" $existingToml) | default (randAlphaNum 64 | quote) }}
{{- if gt (int $expiresAfterSeconds.volumeRead) 0 }}
expires_after_seconds = {{ int $expiresAfterSeconds.volumeRead }}
{{- end }}
@@ -54,7 +59,7 @@ data:
# - the Filer server validates the JWT on writing
# the jwt defaults to expire after 10 seconds
[jwt.filer_signing]
key = {{ include "seaweedfs.existingTomlKey" (list "jwt.filer_signing" $existingToml) | default (randAlphaNum 10 | b64enc | quote) }}
key = {{ include "seaweedfs.existingTomlKey" (list "jwt.filer_signing" $existingToml) | default (randAlphaNum 64 | quote) }}
{{- if gt (int $expiresAfterSeconds.filerWrite) 0 }}
expires_after_seconds = {{ int $expiresAfterSeconds.filerWrite }}
{{- end }}
@@ -66,7 +71,7 @@ data:
# - the Filer server validates the JWT on reading
# the jwt defaults to expire after 60 seconds
[jwt.filer_signing.read]
key = {{ include "seaweedfs.existingTomlKey" (list "jwt.filer_signing.read" $existingToml) | default (randAlphaNum 10 | b64enc | quote) }}
key = {{ include "seaweedfs.existingTomlKey" (list "jwt.filer_signing.read" $existingToml) | default (randAlphaNum 64 | quote) }}
{{- if gt (int $expiresAfterSeconds.filerRead) 0 }}
expires_after_seconds = {{ int $expiresAfterSeconds.filerRead }}
{{- end }}
@@ -121,13 +126,25 @@ data:
cert = "/usr/local/share/ca-certificates/client/tls.crt"
key = "/usr/local/share/ca-certificates/client/tls.key"
# volume server https options
# Note: work in progress!
# this does not work with other clients, e.g., "weed filer|mount" etc, yet.
{{- if include "seaweedfs.volumeHttps" . }}
[https.client]
enabled = true
cert = "/usr/local/share/ca-certificates/client/tls.crt"
key = "/usr/local/share/ca-certificates/client/tls.key"
ca = "/usr/local/share/ca-certificates/ca/tls.crt"
[https.volume]
cert = "/usr/local/share/ca-certificates/volume/tls.crt"
key = "/usr/local/share/ca-certificates/volume/tls.key"
{{- else }}
# volume server https: set global.seaweedfs.volumeHttps.enabled
[https.client]
enabled = false
[https.volume]
cert = ""
key = ""
{{- end }}
{{- end }}
{{- end }}
@@ -25,6 +25,7 @@ spec:
annotations:
{{- with .Values.podAnnotations }}{{ toYaml . | nindent 8 }}{{ end }}
{{- with .Values.vector.podAnnotations }}{{ toYaml . | nindent 8 }}{{ end }}
{{- with include "seaweedfs.volumeHttpsAnnotation" $ }}{{ . | nindent 8 }}{{ end }}
spec:
{{- include "seaweedfs.imagePullSecrets" . | nindent 6 }}
automountServiceAccountToken: {{ .Values.global.seaweedfs.automountServiceAccountToken }}
@@ -150,8 +151,8 @@ spec:
emptyDir: {}
{{- if include "seaweedfs.securityConfigEnabled" . }}
- name: security-config
configMap:
name: {{ include "seaweedfs.fullname" . }}-security-config
secret:
secretName: {{ include "seaweedfs.fullname" . }}-security-config
{{- end }}
{{- if .Values.global.seaweedfs.enableSecurity }}
{{- range list "ca" "client" "vector" }}
@@ -1,6 +1,6 @@
{{- $volumeEnabled := or .Values.volume.enabled .Values.allInOne.enabled }}
{{- if and $volumeEnabled .Values.volume.httpRoute.enabled }}
{{- $serviceName := ternary (include "seaweedfs.componentName" (list . "all-in-one")) (include "seaweedfs.componentName" (list . "volume")) .Values.allInOne.enabled }}
{{- $serviceName := ternary (include "seaweedfs.componentName" (list . "all-in-one")) (include "seaweedfs.componentName" (list . "client-volume")) .Values.allInOne.enabled }}
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
@@ -2,7 +2,7 @@
{{- $volumeEnabled := or .Values.volume.enabled .Values.allInOne.enabled }}
{{- if and $volumeEnabled .Values.volume.ingress.enabled }}
{{- /* Determine service name based on deployment mode */}}
{{- $serviceName := ternary (include "seaweedfs.componentName" (list . "all-in-one")) (include "seaweedfs.componentName" (list . "volume")) .Values.allInOne.enabled }}
{{- $serviceName := ternary (include "seaweedfs.componentName" (list . "all-in-one")) (include "seaweedfs.componentName" (list . "client-volume")) .Values.allInOne.enabled }}
{{- if semverCompare ">=1.19-0" .Capabilities.KubeVersion.GitVersion }}
apiVersion: networking.k8s.io/v1
{{- else if semverCompare ">=1.14-0" .Capabilities.KubeVersion.GitVersion }}
@@ -0,0 +1,42 @@
{{- /* Ready-only backend for the volume ingress and HTTPRoute; the volume Service publishes not-ready pods for DNS.
Not named volume-client: every volume-<key> name belongs to a group under .Values.volumes. */}}
{{- if and .Values.volume.enabled (not .Values.allInOne.enabled) (or .Values.volume.ingress.enabled .Values.volume.httpRoute.enabled) }}
{{- $name := include "seaweedfs.componentName" (list . "client-volume") }}
{{- $volumes := deepCopy .Values.volumes | mergeOverwrite (dict "" .Values.volume) }}
{{- range $vname, $volume := $volumes }}
{{- $volumeName := trimSuffix "-" (printf "volume-%s" $vname) }}
{{- $volume := mergeOverwrite (deepCopy $.Values.volume) (dict "enabled" true) $volume }}
{{- if and $volume.enabled (eq (include "seaweedfs.componentName" (list $ $volumeName)) $name) }}
{{- fail (printf "The volume Ingress/HTTPRoute Service %q has the same name as the %s Service after truncation. Shorten fullnameOverride or the release name." $name $volumeName) }}
{{- end }}
{{- end }}
apiVersion: v1
kind: Service
metadata:
name: {{ $name }}
namespace: {{ include "seaweedfs.namespace" . | quote }}
labels:
app.kubernetes.io/name: {{ template "seaweedfs.name" . }}
app.kubernetes.io/instance: {{ .Release.Name }}
app.kubernetes.io/component: volume
helm.sh/chart: {{ .Chart.Name }}-{{ .Chart.Version | replace "+" "_" }}
app.kubernetes.io/managed-by: {{ .Release.Service }}
{{- with include "seaweedfs.commonLabels" $ }}{{ . | nindent 4 }}{{ end }}
{{- if .Values.volume.annotations }}
annotations:
{{- toYaml .Values.volume.annotations | nindent 4 }}
{{- end }}
spec:
clusterIP: None
internalTrafficPolicy: {{ .Values.volume.internalTrafficPolicy | default "Cluster" }}
{{- /* No metrics port, so the volume ServiceMonitor does not scrape each pod twice */}}
ports:
- name: "swfs-volume"
port: {{ .Values.volume.port }}
targetPort: {{ .Values.volume.port }}
protocol: TCP
selector:
app.kubernetes.io/name: {{ template "seaweedfs.name" . }}
app.kubernetes.io/instance: {{ .Release.Name }}
app.kubernetes.io/component: volume
{{- end }}
@@ -23,6 +23,9 @@ metadata:
{{- end }}
spec:
clusterIP: None
{{- /* The master hands out ${POD_NAME}.<this service> as soon as a volume server heartbeats, before it is Ready.
Ingress and HTTPRoute use the client-volume Service instead, which publishes only Ready pods. */}}
publishNotReadyAddresses: true
internalTrafficPolicy: {{ $volume.internalTrafficPolicy | default "Cluster" }}
ports:
- name: "swfs-volume"
@@ -55,6 +55,7 @@ spec:
{{- with $volume.podAnnotations }}
{{- toYaml . | nindent 8 }}
{{- end }}
{{- with include "seaweedfs.volumeHttpsAnnotation" $ }}{{ . | nindent 8 }}{{ end }}
spec:
{{- if $volume.affinity }}
affinity:
@@ -281,7 +282,7 @@ spec:
httpGet:
path: {{ $volume.readinessProbe.httpGet.path }}
port: {{ $volume.port }}
scheme: {{ $volume.readinessProbe.httpGet.scheme }}
scheme: {{ if include "seaweedfs.volumeHttps" $ }}HTTPS{{ else }}{{ $volume.readinessProbe.httpGet.scheme }}{{ end }}
initialDelaySeconds: {{ $volume.readinessProbe.initialDelaySeconds }}
periodSeconds: {{ $volume.readinessProbe.periodSeconds }}
successThreshold: {{ $volume.readinessProbe.successThreshold }}
@@ -293,7 +294,7 @@ spec:
httpGet:
path: {{ $volume.livenessProbe.httpGet.path }}
port: {{ $volume.port }}
scheme: {{ $volume.livenessProbe.httpGet.scheme }}
scheme: {{ if include "seaweedfs.volumeHttps" $ }}HTTPS{{ else }}{{ $volume.livenessProbe.httpGet.scheme }}{{ end }}
initialDelaySeconds: {{ $volume.livenessProbe.initialDelaySeconds }}
periodSeconds: {{ $volume.livenessProbe.periodSeconds }}
successThreshold: {{ $volume.livenessProbe.successThreshold }}
@@ -370,8 +371,8 @@ spec:
{{- end }}
{{- if include "seaweedfs.securityConfigEnabled" $ }}
- name: security-config
configMap:
name: {{ include "seaweedfs.fullname" $ }}-security-config
secret:
secretName: {{ include "seaweedfs.fullname" $ }}-security-config
{{- end }}
{{- if $.Values.global.seaweedfs.enableSecurity }}
- name: ca-cert
@@ -46,6 +46,7 @@ spec:
{{- with .Values.worker.podAnnotations }}
{{- toYaml . | nindent 8 }}
{{- end }}
{{- with include "seaweedfs.volumeHttpsAnnotation" $ }}{{ . | nindent 8 }}{{ end }}
spec:
restartPolicy: {{ default .Values.global.seaweedfs.restartPolicy .Values.worker.restartPolicy }}
{{- if .Values.worker.affinity }}
@@ -374,8 +375,8 @@ spec:
{{- end }}
{{- if include "seaweedfs.securityConfigEnabled" . }}
- name: security-config
configMap:
name: {{ include "seaweedfs.fullname" . }}-security-config
secret:
secretName: {{ include "seaweedfs.fullname" . }}-security-config
{{- end }}
{{- if .Values.global.seaweedfs.enableSecurity }}
- name: ca-cert
@@ -0,0 +1,107 @@
package tests
import (
"fmt"
"os/exec"
"strings"
"testing"
)
func findDocument(documents []map[string]interface{}, kind, name string) map[string]interface{} {
for _, document := range documents {
if document["kind"] == kind && document["metadata"].(map[string]interface{})["name"] == name {
return document
}
}
return nil
}
func TestVolumeServicePublishesNotReadyPods(t *testing.T) {
_, documents := render(t)
service := findDocument(documents, "Service", "vector-test-seaweedfs-volume")
if service == nil {
t.Fatal("no volume service")
}
if service["spec"].(map[string]interface{})["publishNotReadyAddresses"] != true {
t.Fatal("volume pod DNS names must resolve before the pods are Ready")
}
if findDocument(documents, "Service", "vector-test-seaweedfs-client-volume") != nil {
t.Fatal("client-volume service rendered without ingress or httpRoute")
}
}
func TestVolumeIngressUsesReadyOnlyService(t *testing.T) {
_, documents := render(t, "volume.ingress.enabled=true", "volume.httpRoute.enabled=true", "volume.metricsPort=9327")
client := findDocument(documents, "Service", "vector-test-seaweedfs-client-volume")
if client == nil {
t.Fatal("no client-volume service")
}
spec := client["spec"].(map[string]interface{})
if _, found := spec["publishNotReadyAddresses"]; found {
t.Fatal("client-volume must publish only Ready pods")
}
for _, port := range spec["ports"].([]interface{}) {
if port.(map[string]interface{})["name"] == "metrics" {
t.Fatal("client-volume must not expose metrics, or the ServiceMonitor scrapes each pod twice")
}
}
ingress := findDocument(documents, "Ingress", "ingress-vector-test-seaweedfs-volume")
if ingress == nil {
t.Fatal("no volume ingress")
}
path := ingress["spec"].(map[string]interface{})["rules"].([]interface{})[0].(map[string]interface{})["http"].(map[string]interface{})["paths"].([]interface{})[0].(map[string]interface{})
if backend := path["backend"].(map[string]interface{})["service"].(map[string]interface{})["name"]; backend != "vector-test-seaweedfs-client-volume" {
t.Fatalf("volume ingress backend is %v", backend)
}
route := findDocument(documents, "HTTPRoute", "vector-test-seaweedfs-volume")
if route == nil {
t.Fatal("no volume httpRoute")
}
rule := route["spec"].(map[string]interface{})["rules"].([]interface{})[0].(map[string]interface{})
if backend := rule["backendRefs"].([]interface{})[0].(map[string]interface{})["name"]; backend != "vector-test-seaweedfs-client-volume" {
t.Fatalf("volume httpRoute backend is %v", backend)
}
}
func TestVolumeRoutingServiceDoesNotCollideWithVolumeGroups(t *testing.T) {
_, documents := render(t, "volume.ingress.enabled=true", "volume.httpRoute.enabled=true", "volumes.client.replicas=1")
seen := map[string]bool{}
for _, document := range documents {
key := fmt.Sprint(document["kind"], "/", document["metadata"].(map[string]interface{})["name"])
if seen[key] {
t.Fatalf("%s rendered twice", key)
}
seen[key] = true
}
if !seen["Service/vector-test-seaweedfs-volume-client"] || !seen["Service/vector-test-seaweedfs-client-volume"] {
t.Fatal("volumes.client group Service and the routing Service must both render")
}
}
func TestVolumeRoutingServiceRejectsTruncatedCollision(t *testing.T) {
if _, err := exec.LookPath("helm"); err != nil {
t.Skip("helm is not installed")
}
fullname := strings.Repeat("a", 49) + "-client"
output, err := exec.Command("helm", "template", "test", "..", "--set", "fullnameOverride="+fullname, "--set", "volume.ingress.enabled=true").CombinedOutput()
if err == nil || !strings.Contains(string(output), "same name as the volume Service") {
t.Fatalf("colliding routing Service accepted: %v %s", err, output)
}
}
func TestVolumeIngressAllInOne(t *testing.T) {
_, documents := render(t, "allInOne.enabled=true", "volume.ingress.enabled=true")
if findDocument(documents, "Service", "vector-test-seaweedfs-client-volume") != nil {
t.Fatal("client-volume service rendered in all-in-one mode")
}
ingress := findDocument(documents, "Ingress", "ingress-vector-test-seaweedfs-volume")
if ingress == nil {
t.Fatal("no volume ingress")
}
path := ingress["spec"].(map[string]interface{})["rules"].([]interface{})[0].(map[string]interface{})["http"].(map[string]interface{})["paths"].([]interface{})[0].(map[string]interface{})
if backend := path["backend"].(map[string]interface{})["service"].(map[string]interface{})["name"]; backend != "vector-test-seaweedfs-all-in-one" {
t.Fatalf("all-in-one volume ingress backend is %v", backend)
}
}
+12
View File
@@ -45,6 +45,18 @@ global:
volumeRead: 0
filerWrite: 0
filerRead: 0
# Serve the volume servers' HTTP port over TLS with the chart's volume
# certificate, and make every chart component reach them over HTTPS
# ([https.volume] and [https.client] in security.toml). Requires
# enableSecurity and is not supported with allInOne. Clients outside the
# chart (weed mount, the CSI driver) need their own [https.client].
# The volume servers do not ask for client certificates, because kubelet
# probes cannot present one; JWT signing keeps authorizing volume access.
# [https.client] also sends the admin UI's HTTP lookups on masters and
# S3 over HTTPS, which their plain listeners refuse, so the UI can miss
# volume public URLs and refuse SSE-S3 vector buckets.
volumeHttps:
enabled: false
# we will use this serviceAccountName for all ClusterRoles/ClusterRoleBindings
serviceAccountName: "seaweedfs"
serviceAccountAnnotations: {}
+1040 -1067
View File
File diff suppressed because it is too large. Load diff

Before

Width:  |  Height:  |  Size: 54 KiB

After

Width:  |  Height:  |  Size: 53 KiB

@@ -156,11 +156,14 @@ public class SeaweedWrite {
entry.clearChunks();
entry.addAllChunks(chunks);
filerClient.getBlockingStub().createEntry(
FilerProto.CreateEntryResponse response = filerClient.getBlockingStub().createEntry(
FilerProto.CreateEntryRequest.newBuilder()
.setDirectory(parentDirectory)
.setEntry(entry)
.build());
if (!Strings.isNullOrEmpty(response.getError())) {
throw new IOException(response.getError());
}
}
}
@@ -262,6 +262,8 @@ message CreateEntryRequest {
// Set on a create a filer forwarded to the entry's ring owner, so the owner
// applies it locally instead of forwarding again. Clients leave it unset.
bool is_moved = 8;
// The store write must be durable before the reply, as for ?fsync=true.
bool fsync = 9;
}
// WriteCondition is the precondition the filer evaluates against the existing
+624 -4
View File
@@ -2302,6 +2302,113 @@
}
],
"pluginVersion": "10.3.1"
},
{
"datasource": {
"type": "prometheus",
"uid": "${DS_PROMETHEUS}"
},
"title": "Writable Volumes per Disk",
"description": "Disks holding more than one writable volume of a collection, and the most writable volumes of a collection on one disk.",
"type": "timeseries",
"id": 262,
"gridPos": {
"h": 8,
"w": 12,
"x": 12,
"y": 69
},
"fieldConfig": {
"defaults": {
"color": {
"mode": "palette-classic"
},
"custom": {
"axisBorderShow": false,
"axisCenteredZero": false,
"axisColorMode": "text",
"axisLabel": "",
"axisPlacement": "auto",
"barAlignment": 0,
"drawStyle": "line",
"fillOpacity": 10,
"gradientMode": "none",
"hideFrom": {
"legend": false,
"tooltip": false,
"viz": false
},
"insertNulls": false,
"lineInterpolation": "linear",
"lineWidth": 1,
"pointSize": 4,
"scaleDistribution": {
"type": "linear"
},
"showPoints": "never",
"spanNulls": false,
"stacking": {
"group": "A",
"mode": "none"
},
"thresholdsStyle": {
"mode": "off"
}
},
"mappings": [],
"unit": "short",
"thresholds": {
"mode": "absolute",
"steps": [
{
"color": "green",
"value": null
}
]
}
},
"overrides": []
},
"options": {
"legend": {
"calcs": [
"lastNotNull",
"max"
],
"displayMode": "table",
"placement": "bottom",
"showLegend": true
},
"tooltip": {
"mode": "multi",
"sort": "desc"
}
},
"targets": [
{
"datasource": {
"type": "prometheus",
"uid": "${DS_PROMETHEUS}"
},
"editorMode": "code",
"expr": "count by (collection) (sum by (cluster, collection, data_center, rack, node, disk_id) (SeaweedFS_master_writable_volumes_per_disk{cluster=~\"$cluster\", state=\"writable\"}) > 1)",
"range": true,
"refId": "A",
"legendFormat": "{{collection}} disks with 2+"
},
{
"datasource": {
"type": "prometheus",
"uid": "${DS_PROMETHEUS}"
},
"editorMode": "code",
"expr": "max by (collection) (sum by (cluster, collection, data_center, rack, node, disk_id) (SeaweedFS_master_writable_volumes_per_disk{cluster=~\"$cluster\", state=\"writable\"}))",
"range": true,
"refId": "B",
"legendFormat": "{{collection}} max on one disk"
}
],
"pluginVersion": "10.3.1"
}
]
},
@@ -3335,6 +3442,102 @@
],
"pluginVersion": "10.3.1"
},
{
"datasource": {
"type": "prometheus",
"uid": "${DS_PROMETHEUS}"
},
"title": "Vacuum Dropped Needles",
"description": "Live needles vacuum dropped because their data could not be read back intact (CRC mismatch, malformed or short record), counted since the volume server started. Any non-zero value means a lost replica copy; the volume server log has each dropped key.",
"type": "timeseries",
"id": 267,
"gridPos": {
"h": 8,
"w": 8,
"x": 16,
"y": 54
},
"fieldConfig": {
"defaults": {
"color": {
"mode": "palette-classic"
},
"custom": {
"axisBorderShow": false,
"axisCenteredZero": false,
"axisColorMode": "text",
"axisLabel": "",
"axisPlacement": "auto",
"barAlignment": 0,
"drawStyle": "line",
"fillOpacity": 10,
"gradientMode": "none",
"hideFrom": {
"legend": false,
"tooltip": false,
"viz": false
},
"insertNulls": false,
"lineInterpolation": "linear",
"lineWidth": 1,
"pointSize": 4,
"scaleDistribution": {
"type": "linear"
},
"showPoints": "never",
"spanNulls": false,
"stacking": {
"group": "A",
"mode": "none"
},
"thresholdsStyle": {
"mode": "off"
}
},
"mappings": [],
"unit": "short",
"thresholds": {
"mode": "absolute",
"steps": [
{
"color": "green",
"value": null
}
]
}
},
"overrides": []
},
"options": {
"legend": {
"calcs": [
"lastNotNull",
"max"
],
"displayMode": "table",
"placement": "bottom",
"showLegend": true
},
"tooltip": {
"mode": "multi",
"sort": "desc"
}
},
"targets": [
{
"datasource": {
"type": "prometheus",
"uid": "${DS_PROMETHEUS}"
},
"editorMode": "code",
"expr": "sum by (instance, collection) (SeaweedFS_volumeServer_vacuuming_dropped_needles{cluster=~\"$cluster\"})",
"range": true,
"refId": "A",
"legendFormat": "{{instance}} {{collection}}"
}
],
"pluginVersion": "10.3.1"
},
{
"datasource": {
"type": "prometheus",
@@ -6629,6 +6832,423 @@
],
"pluginVersion": "10.3.1"
},
{
"datasource": {
"type": "prometheus",
"uid": "${DS_PROMETHEUS}"
},
"title": "Metadata Subscribe Delivery Lag",
"description": "Seconds from a metadata event's mutation until the filer sends it to a subscriber, by stream scope",
"type": "timeseries",
"id": 263,
"gridPos": {
"h": 8,
"w": 12,
"x": 0,
"y": 71
},
"fieldConfig": {
"defaults": {
"color": {
"mode": "palette-classic"
},
"custom": {
"axisBorderShow": false,
"axisCenteredZero": false,
"axisColorMode": "text",
"axisLabel": "",
"axisPlacement": "auto",
"barAlignment": 0,
"drawStyle": "line",
"fillOpacity": 10,
"gradientMode": "none",
"hideFrom": {
"legend": false,
"tooltip": false,
"viz": false
},
"insertNulls": false,
"lineInterpolation": "linear",
"lineWidth": 1,
"pointSize": 4,
"scaleDistribution": {
"type": "linear"
},
"showPoints": "never",
"spanNulls": false,
"stacking": {
"group": "A",
"mode": "none"
},
"thresholdsStyle": {
"mode": "off"
}
},
"mappings": [],
"unit": "s",
"thresholds": {
"mode": "absolute",
"steps": [
{
"color": "green",
"value": null
}
]
}
},
"overrides": []
},
"options": {
"legend": {
"calcs": [
"lastNotNull",
"max"
],
"displayMode": "table",
"placement": "bottom",
"showLegend": true
},
"tooltip": {
"mode": "multi",
"sort": "desc"
}
},
"targets": [
{
"datasource": {
"type": "prometheus",
"uid": "${DS_PROMETHEUS}"
},
"editorMode": "code",
"expr": "histogram_quantile(0.99, sum by (le, scope) (rate(SeaweedFS_filer_subscribe_delivery_lag_seconds_bucket{cluster=~\"$cluster\"}[$__rate_interval])))",
"range": true,
"refId": "A",
"legendFormat": "p99 {{scope}}"
},
{
"datasource": {
"type": "prometheus",
"uid": "${DS_PROMETHEUS}"
},
"editorMode": "code",
"expr": "histogram_quantile(0.5, sum by (le, scope) (rate(SeaweedFS_filer_subscribe_delivery_lag_seconds_bucket{cluster=~\"$cluster\"}[$__rate_interval])))",
"range": true,
"refId": "B",
"legendFormat": "p50 {{scope}}"
}
],
"pluginVersion": "10.3.1"
},
{
"datasource": {
"type": "prometheus",
"uid": "${DS_PROMETHEUS}"
},
"title": "Metadata Watermark Hold Duration",
"description": "How long an aggregated read stayed held at one entry until the peers' watermark released it (memory: delivery watermark, disk: flush watermark)",
"type": "timeseries",
"id": 264,
"gridPos": {
"h": 8,
"w": 12,
"x": 12,
"y": 71
},
"fieldConfig": {
"defaults": {
"color": {
"mode": "palette-classic"
},
"custom": {
"axisBorderShow": false,
"axisCenteredZero": false,
"axisColorMode": "text",
"axisLabel": "",
"axisPlacement": "auto",
"barAlignment": 0,
"drawStyle": "line",
"fillOpacity": 10,
"gradientMode": "none",
"hideFrom": {
"legend": false,
"tooltip": false,
"viz": false
},
"insertNulls": false,
"lineInterpolation": "linear",
"lineWidth": 1,
"pointSize": 4,
"scaleDistribution": {
"type": "linear"
},
"showPoints": "never",
"spanNulls": false,
"stacking": {
"group": "A",
"mode": "none"
},
"thresholdsStyle": {
"mode": "off"
}
},
"mappings": [],
"unit": "s",
"thresholds": {
"mode": "absolute",
"steps": [
{
"color": "green",
"value": null
}
]
}
},
"overrides": []
},
"options": {
"legend": {
"calcs": [
"lastNotNull",
"max"
],
"displayMode": "table",
"placement": "bottom",
"showLegend": true
},
"tooltip": {
"mode": "multi",
"sort": "desc"
}
},
"targets": [
{
"datasource": {
"type": "prometheus",
"uid": "${DS_PROMETHEUS}"
},
"editorMode": "code",
"expr": "histogram_quantile(0.99, sum by (le, scope) (rate(SeaweedFS_filer_subscribe_hold_seconds_bucket{cluster=~\"$cluster\"}[$__rate_interval])))",
"range": true,
"refId": "A",
"legendFormat": "p99 {{scope}}"
},
{
"datasource": {
"type": "prometheus",
"uid": "${DS_PROMETHEUS}"
},
"editorMode": "code",
"expr": "histogram_quantile(0.5, sum by (le, scope) (rate(SeaweedFS_filer_subscribe_hold_seconds_bucket{cluster=~\"$cluster\"}[$__rate_interval])))",
"range": true,
"refId": "B",
"legendFormat": "p50 {{scope}}"
}
],
"pluginVersion": "10.3.1"
},
{
"datasource": {
"type": "prometheus",
"uid": "${DS_PROMETHEUS}"
},
"title": "Metadata Delivery Claim / Peer Low-Watermark Age",
"description": "Seconds the filer's delivery claim and the peers' delivery and flush low-watermarks trail the clock; -1 while a tracked peer has not signalled",
"type": "timeseries",
"id": 265,
"gridPos": {
"h": 8,
"w": 12,
"x": 0,
"y": 79
},
"fieldConfig": {
"defaults": {
"color": {
"mode": "palette-classic"
},
"custom": {
"axisBorderShow": false,
"axisCenteredZero": false,
"axisColorMode": "text",
"axisLabel": "",
"axisPlacement": "auto",
"barAlignment": 0,
"drawStyle": "line",
"fillOpacity": 10,
"gradientMode": "none",
"hideFrom": {
"legend": false,
"tooltip": false,
"viz": false
},
"insertNulls": false,
"lineInterpolation": "linear",
"lineWidth": 1,
"pointSize": 4,
"scaleDistribution": {
"type": "linear"
},
"showPoints": "never",
"spanNulls": false,
"stacking": {
"group": "A",
"mode": "none"
},
"thresholdsStyle": {
"mode": "off"
}
},
"mappings": [],
"unit": "s",
"thresholds": {
"mode": "absolute",
"steps": [
{
"color": "green",
"value": null
}
]
}
},
"overrides": []
},
"options": {
"legend": {
"calcs": [
"lastNotNull",
"max"
],
"displayMode": "table",
"placement": "bottom",
"showLegend": true
},
"tooltip": {
"mode": "multi",
"sort": "desc"
}
},
"targets": [
{
"datasource": {
"type": "prometheus",
"uid": "${DS_PROMETHEUS}"
},
"editorMode": "code",
"expr": "max by (instance) (SeaweedFS_filer_meta_delivery_claim_age_seconds{cluster=~\"$cluster\"})",
"range": true,
"refId": "A",
"legendFormat": "claim {{instance}}"
},
{
"datasource": {
"type": "prometheus",
"uid": "${DS_PROMETHEUS}"
},
"editorMode": "code",
"expr": "max by (instance, kind) (SeaweedFS_filer_meta_peer_low_watermark_age_seconds{cluster=~\"$cluster\"})",
"range": true,
"refId": "B",
"legendFormat": "peer low {{kind}} {{instance}}"
}
],
"pluginVersion": "10.3.1"
},
{
"datasource": {
"type": "prometheus",
"uid": "${DS_PROMETHEUS}"
},
"title": "Metadata Subscribers Resumed from Disk",
"description": "Times a metadata subscriber fell off the in-memory ring and resumed from the persisted log",
"type": "timeseries",
"id": 266,
"gridPos": {
"h": 8,
"w": 12,
"x": 12,
"y": 79
},
"fieldConfig": {
"defaults": {
"color": {
"mode": "palette-classic"
},
"custom": {
"axisBorderShow": false,
"axisCenteredZero": false,
"axisColorMode": "text",
"axisLabel": "",
"axisPlacement": "auto",
"barAlignment": 0,
"drawStyle": "line",
"fillOpacity": 10,
"gradientMode": "none",
"hideFrom": {
"legend": false,
"tooltip": false,
"viz": false
},
"insertNulls": false,
"lineInterpolation": "linear",
"lineWidth": 1,
"pointSize": 4,
"scaleDistribution": {
"type": "linear"
},
"showPoints": "never",
"spanNulls": false,
"stacking": {
"group": "A",
"mode": "none"
},
"thresholdsStyle": {
"mode": "off"
}
},
"mappings": [],
"unit": "ops",
"thresholds": {
"mode": "absolute",
"steps": [
{
"color": "green",
"value": null
}
]
}
},
"overrides": []
},
"options": {
"legend": {
"calcs": [
"lastNotNull",
"max"
],
"displayMode": "table",
"placement": "bottom",
"showLegend": true
},
"tooltip": {
"mode": "multi",
"sort": "desc"
}
},
"targets": [
{
"datasource": {
"type": "prometheus",
"uid": "${DS_PROMETHEUS}"
},
"editorMode": "code",
"expr": "sum by (instance, scope) (rate(SeaweedFS_filer_subscribe_resume_from_disk{cluster=~\"$cluster\"}[$__rate_interval]))",
"range": true,
"refId": "A",
"legendFormat": "{{scope}} {{instance}}"
}
],
"pluginVersion": "10.3.1"
},
{
"datasource": {
"type": "prometheus",
@@ -6642,7 +7262,7 @@
"h": 8,
"w": 12,
"x": 0,
"y": 71
"y": 87
},
"fieldConfig": {
"defaults": {
@@ -6738,7 +7358,7 @@
"h": 8,
"w": 12,
"x": 12,
"y": 71
"y": 87
},
"fieldConfig": {
"defaults": {
@@ -6834,7 +7454,7 @@
"h": 8,
"w": 12,
"x": 0,
"y": 79
"y": 95
},
"fieldConfig": {
"defaults": {
@@ -6930,7 +7550,7 @@
"h": 8,
"w": 12,
"x": 12,
"y": 79
"y": 95
},
"fieldConfig": {
"defaults": {
+6
View File
@@ -7143,8 +7143,10 @@ dependencies = [
"aes",
"aes-gcm",
"anyhow",
"arrow-array",
"arrow-cast",
"arrow-schema",
"arrow-select",
"async-trait",
"base64 0.22.1",
"bytes",
@@ -7158,6 +7160,7 @@ dependencies = [
"hmac 0.12.1",
"http",
"hyper-util",
"jsonb",
"jsonwebtoken",
"lance",
"lance-datafusion",
@@ -7190,6 +7193,7 @@ dependencies = [
"tower",
"tracing",
"tracing-subscriber",
"uuid",
"x509-parser",
"zstd",
]
@@ -7208,6 +7212,7 @@ dependencies = [
"lance",
"lance-index",
"lance-linalg",
"lance-select",
"lance-table",
"object_store",
"roaring",
@@ -7236,6 +7241,7 @@ dependencies = [
"futures",
"lance",
"lance-arrow",
"lance-index",
"lance-linalg",
"moka",
"object_store",
+1
View File
@@ -22,6 +22,7 @@ object_store = "0.13"
lance-index = "10"
lance-linalg = "10"
lance-table = "10"
lance-select = "10"
arrow-array = "58"
arrow-cast = "58"
arrow-schema = "58"
@@ -139,6 +139,7 @@ mod tests {
nprobes: Some(4),
index_segments: None,
fragments: None,
exact_match_limit: None,
},
)
.await
@@ -167,6 +168,7 @@ mod tests {
nprobes: Some(4),
index_segments: None,
fragments: None,
exact_match_limit: None,
};
let filtered = ann(&dataset, &table, &query).await.unwrap();
let exact = flat(&dataset, &table, &query).await.unwrap();
@@ -204,6 +206,7 @@ mod tests {
nprobes: None,
index_segments: None,
fragments: None,
exact_match_limit: None,
},
)
.await
+17 -7
View File
@@ -4,6 +4,7 @@
//! compares the elements as typed JSON values: Lance's compares their text, so
//! 1 would match "1".
use std::collections::HashMap;
use std::sync::Arc;
use arrow_array::builder::{BooleanBuilder, StringBuilder};
@@ -19,13 +20,21 @@ use lance::deps::datafusion::logical_expr::{
ColumnarValue, Expr, ScalarFunctionArgs, ScalarUDF, ScalarUDFImpl, Signature, Volatility,
};
/// Set `filter` on the scanner with the JSONPath functions swapped.
pub(crate) fn apply(scanner: &mut Scanner, filter: &str) -> anyhow::Result<()> {
/// Set `filter` on the scanner with the JSONPath functions swapped and the
/// conjuncts on `ready` indexed keys answered by their columns, which it
/// returns.
pub(crate) fn apply(
scanner: &mut Scanner,
filter: &str,
ready: &HashMap<String, String>,
) -> anyhow::Result<Vec<crate::indexed_filter::Indexed>> {
scanner.filter(filter)?;
if let Some(expr) = scanner.get_expr_filter()? {
scanner.filter_expr(swap(expr)?);
}
Ok(())
let Some(expr) = scanner.get_expr_filter()? else {
return Ok(Vec::new());
};
let (expr, indexed) = crate::indexed_filter::rewrite(expr, ready);
scanner.filter_expr(swap(expr)?);
Ok(indexed)
}
fn swap(expr: Expr) -> Result<Expr> {
@@ -268,7 +277,7 @@ mod tests {
async fn keys(dataset: &Dataset, filter: &str, swapped: bool) -> anyhow::Result<Vec<String>> {
let mut scanner = dataset.scan();
if swapped {
apply(&mut scanner, filter)?;
apply(&mut scanner, filter, &HashMap::new())?;
} else {
scanner.filter(filter)?;
}
@@ -395,6 +404,7 @@ mod tests {
apply(
&mut scanner,
r#"json_extract(`metadata`, '$["k"]') = '"v"' AND json_exists(`metadata`, '$["k"]') AND json_get_string(`metadata`, 'k') = 'v' AND json_array_contains(`metadata`, '$["k"]', '"v"') AND NOT json_array_contains(`metadata`, '$["k"]', key)"#,
&HashMap::new(),
)
.unwrap();
let mut swapped = Vec::new();
+78
View File
@@ -104,6 +104,7 @@ mod tests {
nprobes: None,
index_segments: None,
fragments: None,
exact_match_limit: None,
}
}
@@ -198,6 +199,83 @@ mod tests {
assert_eq!(filtered.len(), 10);
}
/// A filtered query through the index returns every eligible row when
/// fewer than k match, though the probed partitions cannot hold them all:
/// 400 of 4000 rows match, the index has 63 partitions and probes at most
/// 32 of them.
#[tokio::test]
async fn filtered_ann_returns_every_match_below_k() {
let dir = TempDir::new().unwrap();
let rows = scattered(4000, 8, 0x9E3779B97F4A7C15);
let tags: Vec<&str> = (0..4000)
.map(|i| if i % 10 == 0 { "rare" } else { "common" })
.collect();
let mut dataset = testing::write_dataset(&dir, &rows, &tags).await;
let table = VectorTable::resolve(&dataset, None, None).unwrap();
build(
&mut dataset,
&table.vector_column,
INDEX_NAME,
partitions_for(4000),
false,
)
.await
.unwrap();
assert_eq!(partitions_for(4000), 63);
for vector in &scattered(3, 8, 0xD1B54A32D192ED03) {
let q = Query {
filter: Some("tag = 'rare'".to_string()),
..exact(vector, 500)
};
let flat = query::flat(&dataset, &table, &q).await.unwrap();
assert_eq!(flat.len(), 400);
let ann = query::ann(&dataset, &table, &q).await.unwrap();
assert_eq!(ann.len(), 400, "the index path dropped eligible rows");
for (f, a) in flat.iter().zip(ann.iter()) {
assert!((f.distance - a.distance).abs() < 1e-5);
}
}
}
/// When no more than k rows match, every match comes back with its own
/// distance, nearest first — not the rows the probes missed with an
/// infinite distance. Ten fragments of 400 rows, 5 of them matching.
#[tokio::test]
async fn filtered_ann_scores_every_match_at_or_below_k() {
let dir = TempDir::new().unwrap();
let rows = scattered(4000, 8, 0x9E3779B97F4A7C15);
let tags: Vec<&str> = (0..4000)
.map(|i| if i % 800 == 0 { "rare" } else { "common" })
.collect();
let mut dataset = testing::write_dataset(&dir, &rows[..400], &tags[..400]).await;
for chunk in 1..10 {
let r = chunk * 400..(chunk + 1) * 400;
testing::append_dataset(&mut dataset, &rows[r.clone()], &tags[r]).await;
}
let table = VectorTable::resolve(&dataset, None, None).unwrap();
build(&mut dataset, &table.vector_column, INDEX_NAME, 63, false)
.await
.unwrap();
for vector in &scattered(3, 8, 0xD1B54A32D192ED03) {
for k in [5, 10] {
let q = Query {
filter: Some("tag = 'rare'".to_string()),
..exact(vector, k)
};
let flat = query::flat(&dataset, &table, &q).await.unwrap();
assert_eq!(flat.len(), 5);
let ann = query::ann(&dataset, &table, &q).await.unwrap();
assert_eq!(ann.len(), 5);
for (f, a) in flat.iter().zip(ann.iter()) {
assert!(
(f.distance - a.distance).abs() < 1e-5,
"k={k}: {flat:?} vs {ann:?}"
);
}
}
}
}
#[tokio::test]
async fn default_probes_take_the_largest_index() {
use lance::index::DatasetIndexExt;
@@ -0,0 +1,739 @@
//! Filters on indexed metadata keys (see `weed_lance_worker::metadata_index`).
//!
//! The gateway renders a string `$eq` or `$in` on a key as an OR of
//! `json_extract(metadata, P) = '"v"'` (or `IN`) and
//! `json_array_contains(metadata, P, '"v"')` over the same values. In the
//! parsed filter, such a top-level conjunct on a ready key is replaced by
//! `array_has` / `array_has_any` on the key's column, which its LABEL_LIST
//! index answers. Everything else, including `$ne` and `$nin`, stays on the
//! JSON path.
use std::collections::{BTreeMap, BTreeSet, HashMap};
use anyhow::Result;
use jsonb::OwnedJsonb;
use lance::Dataset;
use lance::deps::datafusion::common::{Column, ScalarValue};
use lance::deps::datafusion::functions_nested::expr_fn::{array_has, array_has_any};
use lance::deps::datafusion::logical_expr::utils::split_conjunction_owned;
use lance::deps::datafusion::logical_expr::{BinaryExpr, Expr, Operator};
use lance::deps::datafusion::prelude::lit;
use weed_lance_worker::metadata_index::{self, State};
const METADATA_COLUMN: &str = "metadata";
/// The column of each ready indexed key of `dataset`, by metadata key.
pub(crate) fn ready_columns(dataset: &Dataset) -> HashMap<String, String> {
let schema: arrow_schema::Schema = dataset.schema().into();
metadata_index::indexed_keys(&schema)
.into_iter()
.filter(|k| k.state == Some(State::Ready))
.map(|k| (k.key, k.column))
.collect()
}
/// A conjunct a key's column answers: rows whose `column` holds one of
/// `values`.
#[derive(Debug, Clone)]
pub(crate) struct Indexed {
pub column: String,
pub values: Vec<String>,
pub expr: Expr,
}
/// `filter` with each top-level conjunct that is a string `$eq` or `$in` on
/// a key in `ready` answered by the key's column instead, and those
/// replacements.
pub(crate) fn rewrite(filter: Expr, ready: &HashMap<String, String>) -> (Expr, Vec<Indexed>) {
if ready.is_empty() {
return (filter, Vec::new());
}
let mut indexed: Vec<Indexed> = Vec::new();
let mut rewritten: Option<Expr> = None;
for conjunct in split_conjunction_owned(filter) {
let replacement = string_membership(&conjunct).and_then(|(key, values)| {
let name = ready.get(&key)?;
let column = Expr::Column(Column::from_name(name));
let expr = match values.as_slice() {
[one] => array_has(column, lit(one.as_str())),
_ => {
let values: Vec<ScalarValue> = values
.iter()
.map(|v| ScalarValue::Utf8(Some(v.clone())))
.collect();
let list = ScalarValue::new_list(&values, &arrow_schema::DataType::Utf8, true);
array_has_any(column, Expr::Literal(ScalarValue::List(list), None))
}
};
Some(Indexed {
column: name.clone(),
values,
expr,
})
});
let part = match replacement {
Some(found) => {
let expr = found.expr.clone();
indexed.push(found);
expr
}
None => conjunct,
};
rewritten = Some(and(rewritten, part));
}
(rewritten.unwrap_or(lit(true)), indexed)
}
fn and(left: Option<Expr>, right: Expr) -> Expr {
match left {
Some(left) => Expr::BinaryExpr(BinaryExpr::new(
Box::new(left),
Operator::And,
Box::new(right),
)),
None => right,
}
}
/// The key and string values of a conjunct meaning "the key holds one of
/// these strings, or an array with one of them": the OR of scalar matches
/// and array-member matches over the same values and key.
fn string_membership(expr: &Expr) -> Option<(String, Vec<String>)> {
let mut disjuncts = Vec::new();
split_or(expr, &mut disjuncts);
let mut path: Option<&str> = None;
let mut scalars = BTreeSet::new();
let mut elements = BTreeSet::new();
for disjunct in disjuncts {
let (p, kind) = match unwrap_false_coalesce(disjunct) {
Expr::BinaryExpr(BinaryExpr {
left,
op: Operator::Eq,
right,
}) => (
json_function(left, "json_extract")?,
Member::Scalar(vec![utf8(right)?]),
),
Expr::InList(list) if !list.negated => (
json_function(&list.expr, "json_extract")?,
Member::Scalar(list.list.iter().map(utf8).collect::<Option<_>>()?),
),
Expr::ScalarFunction(f) if f.func.name() == "json_array_contains" => {
let [data, p, value] = f.args.as_slice() else {
return None;
};
if !is_metadata(data) {
return None;
}
let element: serde_json::Value = serde_json::from_str(utf8(value)?).ok()?;
(utf8(p)?, Member::Element(element.as_str()?.to_string()))
}
_ => return None,
};
if path.is_some_and(|q| q != p) {
return None;
}
path = Some(p);
match kind {
Member::Scalar(values) => scalars.extend(values.into_iter().map(str::to_string)),
Member::Element(value) => {
elements.insert(value);
}
}
}
let key = json_path_key(path?)?;
let rendered: BTreeSet<String> = elements
.iter()
.map(|s| jsonb_text(s))
.collect::<Option<_>>()?;
(!elements.is_empty() && rendered == scalars).then(|| (key, elements.into_iter().collect()))
}
enum Member<'a> {
Scalar(Vec<&'a str>),
Element(String),
}
fn split_or<'a>(expr: &'a Expr, out: &mut Vec<&'a Expr>) {
match expr {
Expr::BinaryExpr(BinaryExpr {
left,
op: Operator::Or,
right,
}) => {
split_or(left, out);
split_or(right, out);
}
Expr::Alias(alias) => split_or(&alias.expr, out),
other => out.push(other),
}
}
/// `x` for `COALESCE(x, false)` or its simplified `x IS NOT NULL AND x`;
/// either is false where `x` is null, which a top-level conjunct treats alike.
fn unwrap_false_coalesce(expr: &Expr) -> &Expr {
match expr {
Expr::ScalarFunction(f) if f.func.name() == "coalesce" => match f.args.as_slice() {
[x, Expr::Literal(ScalarValue::Boolean(Some(false)), _)] => x,
_ => expr,
},
Expr::BinaryExpr(BinaryExpr {
left,
op: Operator::And,
right,
}) => match left.as_ref() {
Expr::IsNotNull(x) if x == right => right,
_ => expr,
},
_ => expr,
}
}
fn utf8(expr: &Expr) -> Option<&str> {
match expr {
Expr::Literal(ScalarValue::Utf8(Some(s)) | ScalarValue::LargeUtf8(Some(s)), _) => {
Some(s.as_str())
}
_ => None,
}
}
fn is_metadata(expr: &Expr) -> bool {
matches!(expr, Expr::Column(c) if c.name == METADATA_COLUMN)
}
/// The JSONPath literal of `name(metadata, path)`.
fn json_function<'a>(expr: &'a Expr, name: &str) -> Option<&'a str> {
match expr {
Expr::ScalarFunction(f) if f.func.name() == name => match f.args.as_slice() {
[data, path] if is_metadata(data) => utf8(path),
_ => None,
},
_ => None,
}
}
/// The key of a `$["key"]` path, with `\\` and `\"` unescaped.
fn json_path_key(path: &str) -> Option<String> {
let inner = path.strip_prefix("$[\"")?.strip_suffix("\"]")?;
let mut key = String::with_capacity(inner.len());
let mut chars = inner.chars();
while let Some(c) = chars.next() {
match c {
'\\' => match chars.next()? {
c @ ('\\' | '"') => key.push(c),
_ => return None,
},
'"' => return None,
c => key.push(c),
}
}
Some(key)
}
/// What json_extract returns for a stored string `s`.
fn jsonb_text(s: &str) -> Option<String> {
let value: OwnedJsonb = serde_json::to_string(s).ok()?.parse().ok()?;
Some(value.to_string())
}
/// Segments searched at once when counting.
const PARALLEL_SEGMENTS: usize = 8;
/// The rows of `scope` (every fragment when None) whose column matches
/// `indexed`, or None once there are more than `limit`. Segments are
/// searched newest first, several at once, and counting stops as soon as
/// the total passes the limit. Rows the index has not reached are counted
/// by a read of the column. Deleted rows an index segment still lists
/// count too: the total is an upper bound, which only ever sends a query
/// to the vector index instead of an exact scan.
pub(crate) async fn count_indexed(
dataset: &Dataset,
indexed: &Indexed,
scope: Option<&[u64]>,
limit: u64,
) -> Result<Option<u64>> {
use futures::{StreamExt, TryStreamExt};
use lance::index::DatasetIndexInternalExt;
use lance_index::metrics::NoOpMetricsCollector;
use lance_index::scalar::{LabelListQuery, SearchResult};
use lance_select::RowSetOps;
let scope: Option<roaring::RoaringBitmap> =
scope.map(|ids| ids.iter().map(|id| *id as u32).collect());
let segments = metadata_index::segments(dataset, &indexed.column).await?;
let query = LabelListQuery::HasAnyLabel(
indexed
.values
.iter()
.map(|v| ScalarValue::Utf8(Some(v.clone())))
.collect(),
);
let mut searches: Vec<futures::future::BoxFuture<'_, Result<u64>>> = Vec::new();
for segment in segments.iter().rev() {
let fragments = match &scope {
Some(scope) => &segment.fragments & scope,
None => segment.fragments.clone(),
};
if fragments.is_empty() {
continue;
}
let uuid = segment.uuid;
let query = &query;
searches.push(Box::pin(async move {
let index = dataset
.open_scalar_index(&indexed.column, &uuid, &NoOpMetricsCollector)
.await?;
let mut found = match index.search(query, &NoOpMetricsCollector).await? {
SearchResult::Exact(rows) | SearchResult::AtMost(rows) => rows.true_rows(),
SearchResult::AtLeast(_) => anyhow::bail!("a LABEL_LIST index answered AtLeast"),
};
found.retain_fragments(fragments.iter());
Ok(found.len().unwrap_or(u64::MAX))
}));
}
let mut counts = futures::stream::iter(searches).buffered(PARALLEL_SEGMENTS);
let mut total = 0u64;
while let Some(count) = counts.try_next().await? {
total = total.saturating_add(count);
if total > limit {
return Ok(None);
}
}
let unindexed: Vec<lance_table::format::Fragment> = dataset
.fragments()
.iter()
.filter(|f| !segments.iter().any(|s| s.fragments.contains(f.id as u32)))
.filter(|f| scope.as_ref().is_none_or(|s| s.contains(f.id as u32)))
.cloned()
.collect();
if !unindexed.is_empty() {
let mut scanner = dataset.scan();
scanner
.filter_expr(indexed.expr.clone())
.with_fragments(unindexed);
total = total.saturating_add(scanner.count_rows().await?);
}
Ok((total <= limit).then_some(total))
}
/// How a filter compares a key.
pub mod operand {
pub const STRING: u8 = 1;
pub const NUMBER: u8 = 2;
pub const BOOL: u8 = 4;
pub const EXISTS: u8 = 8;
}
/// The metadata keys `filter` reads, each with the [`operand`] kinds it is
/// compared with.
pub fn filter_keys(dataset: &Dataset, filter: &str) -> Result<BTreeMap<String, u8>> {
let mut scanner = dataset.scan();
scanner.filter(filter)?;
let mut keys = BTreeMap::new();
if let Some(expr) = scanner.get_expr_filter()? {
collect_keys(&expr, &mut keys);
}
Ok(keys)
}
fn literal_kind(text: &str) -> u8 {
match serde_json::from_str::<serde_json::Value>(text) {
Ok(serde_json::Value::String(_)) => operand::STRING,
Ok(serde_json::Value::Number(_)) => operand::NUMBER,
Ok(serde_json::Value::Bool(_)) => operand::BOOL,
// json_extract's text of a string with control characters is not
// JSON, but it is quoted.
_ if text.starts_with('"') => operand::STRING,
_ => 0,
}
}
fn collect_keys(expr: &Expr, keys: &mut BTreeMap<String, u8>) {
let mut note = |path: &str, kind: u8| {
if let Some(key) = json_path_key(path) {
*keys.entry(key).or_default() |= kind;
}
};
match expr {
Expr::TryCast(cast) if extract(&cast.expr).is_some() => {
note(extract(&cast.expr).unwrap_or_default(), operand::NUMBER);
return;
}
Expr::BinaryExpr(BinaryExpr { left, right, .. }) => {
if let (Some(path), Some(value)) = (extract(left), utf8(right)) {
note(path, literal_kind(value));
return;
}
}
Expr::InList(list) => {
if let Some(path) = extract(&list.expr) {
let kind = list
.list
.iter()
.filter_map(utf8)
.fold(0, |k, v| k | literal_kind(v));
note(path, kind);
return;
}
}
Expr::ScalarFunction(f) if is_json_function(f.func.name()) => {
if let [data, path, rest @ ..] = f.args.as_slice()
&& is_metadata(data)
&& let Some(path) = utf8(path)
{
let kind = match (f.func.name(), rest) {
("json_exists", _) => operand::EXISTS,
(_, [value]) => utf8(value).map_or(0, literal_kind),
_ => 0,
};
note(path, kind);
return;
}
}
_ => {}
}
for child in children(expr) {
collect_keys(child, keys);
}
}
fn extract(expr: &Expr) -> Option<&str> {
json_function(expr, "json_extract")
}
fn is_json_function(name: &str) -> bool {
matches!(name, "json_extract" | "json_exists" | "json_array_contains")
}
fn children(expr: &Expr) -> Vec<&Expr> {
use lance::deps::datafusion::common::tree_node::TreeNode;
let mut out = Vec::new();
let _ = expr.apply_children(|child| {
out.push(child);
Ok(lance::deps::datafusion::common::tree_node::TreeNodeRecursion::Continue)
});
out
}
#[cfg(test)]
mod tests {
use super::*;
use crate::query::{self, Query};
use crate::table::VectorTable;
use arrow_array::{FixedSizeListArray, RecordBatch, RecordBatchIterator, StringArray};
use arrow_schema::{DataType, Field, Schema};
use lance::index::DatasetIndexExt;
use lance_index::IndexType;
use lance_index::scalar::{BuiltinIndexType, ScalarIndexParams};
use lance_linalg::distance::DistanceType;
use std::sync::Arc;
use tempfile::TempDir;
const DIM: usize = 4;
/// The gateway's text for `{"author": "<v>"}`.
fn eq(field: &str, json: &str) -> String {
let json = json.replace('\'', "''");
format!(
"(COALESCE(json_extract(`metadata`, '$[\"{field}\"]') = '{json}', false) OR COALESCE(json_array_contains(`metadata`, '$[\"{field}\"]', '{json}'), false))"
)
}
/// The gateway's text for `{"author": {"$in": [<strings>]}}`.
fn among(values: &[&str]) -> String {
let scalars: Vec<String> = values.iter().map(|v| format!("'\"{v}\"'")).collect();
let mut groups = vec![format!(
"COALESCE(json_extract(`metadata`, '$[\"author\"]') in ({}), false)",
scalars.join(", ")
)];
for v in values {
groups.push(format!(
"COALESCE(json_array_contains(`metadata`, '$[\"author\"]', '\"{v}\"'), false)"
));
}
format!("({})", groups.join(" OR "))
}
/// Rows `u0..u<n>` by author, plus a few others, in fragments of 100,
/// with the author column ready and indexed and a vector index.
async fn dataset(dir: &TempDir, rows: usize) -> Dataset {
let metadata = Field::new("metadata", DataType::Utf8, true)
.with_metadata([("ARROW:extension:name".to_string(), "arrow.json".to_string())].into());
let schema = Arc::new(Schema::new(vec![
Field::new("key", DataType::Utf8, false),
Field::new(
"vector",
DataType::FixedSizeList(
Arc::new(Field::new("item", DataType::Float32, true)),
DIM as i32,
),
false,
),
metadata,
metadata_index::field("_mk_0", "author", State::Ready),
]));
let metas: Vec<serde_json::Value> = (0..rows)
.map(|i| match i % 10 {
0 => serde_json::json!({"author": "rare"}),
1 => serde_json::json!({"author": ["many", "rare2"]}),
_ => serde_json::json!({"author": "many", "n": i}),
})
.collect();
let batches: Vec<RecordBatch> = (0..rows)
.step_by(100)
.map(|start| {
let range = start..(start + 100).min(rows);
RecordBatch::try_new(
schema.clone(),
vec![
Arc::new(StringArray::from(
range.clone().map(|i| format!("k{i}")).collect::<Vec<_>>(),
)),
Arc::new(FixedSizeListArray::from_iter_primitive::<
arrow_array::types::Float32Type,
_,
_,
>(
range.clone().map(|i| {
Some((0..DIM).map(move |j| Some(((i * 31 + j * 17) % 97) as f32)))
}),
DIM as i32,
)),
Arc::new(StringArray::from(
range
.clone()
.map(|i| Some(metas[i].to_string()))
.collect::<Vec<_>>(),
)),
metadata_index::column_from_json(range.map(|i| Some(&metas[i])), "author"),
],
)
.unwrap()
})
.collect();
let uri = dir.path().to_str().unwrap();
let mut dataset = None::<Dataset>;
for batch in batches {
let reader = RecordBatchIterator::new(vec![Ok(batch)], schema.clone());
dataset = Some(match dataset {
None => Dataset::write(reader, uri, None).await.unwrap(),
Some(mut d) => {
d.append(reader, None).await.unwrap();
d
}
});
}
let mut dataset = dataset.unwrap();
dataset
.create_index(
&["_mk_0"],
IndexType::LabelList,
None,
&ScalarIndexParams::for_builtin(BuiltinIndexType::LabelList),
true,
)
.await
.unwrap();
crate::index::build(&mut dataset, "vector", crate::index::INDEX_NAME, 4, false)
.await
.unwrap();
dataset
}
fn rewritten(dataset: &Dataset, filter: &str) -> (String, Option<String>) {
let mut scanner = dataset.scan();
let indexed = crate::filter::apply(&mut scanner, filter, &ready_columns(dataset)).unwrap();
(
scanner.get_expr_filter().unwrap().unwrap().to_string(),
indexed.first().map(|i| i.expr.to_string()),
)
}
#[tokio::test]
async fn string_eq_and_in_on_a_ready_key_use_its_column() {
let dir = TempDir::new().unwrap();
let dataset = dataset(&dir, 200).await;
assert_eq!(
ready_columns(&dataset),
HashMap::from([("author".to_string(), "_mk_0".to_string())])
);
for (filter, want) in [
(eq("author", "\"rare\""), Some("array_has(_mk_0, Utf8(\"rare\"))")),
(
among(&["rare", "many"]),
Some("array_has_any(_mk_0, List([many, rare]))"),
),
(
format!("{} AND TRY_CAST(json_extract(`metadata`, '$[\"n\"]') AS DOUBLE) > 3", eq("author", "\"rare\"")),
Some("array_has(_mk_0, Utf8(\"rare\"))"),
),
(eq("author", "\"o'neil \\\"q\\\"\""), Some("array_has(_mk_0, Utf8(\"o'neil \"q\"\"))")),
(
"(COALESCE(json_extract(`metadata`, '$[\"author\"]') = '\"c\\u0001d\\n\"', false) OR COALESCE(json_array_contains(`metadata`, '$[\"author\"]', '\"c\\u0001d\\n\"'), false))".to_string(),
Some("array_has(_mk_0, Utf8(\"c\u{1}d\n\"))"),
),
// The scalar half's text is not what json_extract returns for
// the array half's string, so the two do not agree.
(
"(COALESCE(json_extract(`metadata`, '$[\"author\"]') = '\"c\\u{1}d\\n\"', false) OR COALESCE(json_array_contains(`metadata`, '$[\"author\"]', '\"c\\u0001d\\n\"'), false))".to_string(),
None,
),
// Not a string, not this key, or not every value on both sides.
(eq("author", "true"), None),
(eq("other", "\"rare\""), None),
(format!("NOT {}", eq("author", "\"rare\"")), None),
(format!("NOT {}", among(&["rare", "many"])), None),
(
"COALESCE(json_extract(`metadata`, '$[\"author\"]') = '\"rare\"', false)".to_string(),
None,
),
(
among(&["rare", "many"]).replace("'\"many\"'), false))", "'\"other\"'), false))"),
None,
),
("json_exists(`metadata`, '$[\"author\"]')".to_string(), None),
(format!("({} OR json_exists(`metadata`, '$[\"n\"]'))", eq("author", "\"rare\"")), None),
] {
let (expr, indexed) = rewritten(&dataset, &filter);
assert_eq!(indexed.as_deref(), want, "{filter}");
assert_eq!(expr.contains("json_"), want.is_none() || filter.contains(" AND "), "{filter}: {expr}");
}
}
/// The planner scores few matches exactly and leaves many to the vector
/// index, per scope.
#[tokio::test]
async fn few_indexed_matches_are_scored_exactly() {
let dir = TempDir::new().unwrap();
let dataset = dataset(&dir, 1000).await;
let table = VectorTable::resolve(&dataset, None, None).unwrap();
let query = |filter: &str, fragments: Option<Vec<u64>>| Query {
vector: vec![1.0, 2.0, 3.0, 4.0],
k: 10,
filter: Some(filter.to_string()),
postfilter: false,
distance: DistanceType::L2,
nprobes: None,
index_segments: None,
fragments,
exact_match_limit: None,
};
let rare = eq("author", "\"rare\"");
let many = eq("author", "\"many\"");
let few = |q: Query| {
let (dataset, table) = (&dataset, &table);
async move {
query::few_indexed_matches(dataset, table, &q)
.await
.unwrap()
}
};
// 100 and 900 matches, and 90 in the first two fragments.
assert!(few(query(&rare, None)).await);
assert!(!few(query(&many, None)).await);
let first = Some(vec![0, 1]);
assert!(few(query(&many, first.clone())).await);
assert!(
!few(Query {
exact_match_limit: Some(0),
..query(&rare, None)
})
.await
);
// A JSON-only filter is not counted.
assert!(!few(query("json_exists(`metadata`, '$[\"n\"]')", None)).await);
for (filter, fragments) in [(&rare, None), (&many, first)] {
let q = query(filter, fragments);
let exact = query::flat(&dataset, &table, &q).await.unwrap();
let got = query::ann(&dataset, &table, &q).await.unwrap();
assert_eq!(got.len(), 10);
for (g, e) in got.iter().zip(&exact) {
assert!(
(g.distance - e.distance).abs() < 1e-4,
"{got:?} vs {exact:?}"
);
}
}
}
#[tokio::test]
async fn the_plan_reads_the_scalar_index() {
let dir = TempDir::new().unwrap();
let dataset = dataset(&dir, 300).await;
let mut scanner = dataset.scan();
crate::filter::apply(
&mut scanner,
&eq("author", "\"rare\""),
&ready_columns(&dataset),
)
.unwrap();
let plan = scanner.explain_plan(true).await.unwrap();
assert!(plan.contains("LabelList"), "{plan}");
assert!(!plan.contains("json_"), "{plan}");
}
#[tokio::test]
async fn filter_keys_note_how_each_key_is_compared() {
let dir = TempDir::new().unwrap();
let dataset = dataset(&dir, 100).await;
let filter = format!(
"{} AND TRY_CAST(json_extract(`metadata`, '$[\"n\"]') AS DOUBLE) > 3 AND NOT json_exists(`metadata`, '$[\"a.b\\\"c\"]') AND COALESCE(json_extract(`metadata`, '$[\"f\"]') = 'true', false)",
eq("author", "\"rare\"")
);
let keys = filter_keys(&dataset, &filter).unwrap();
assert_eq!(
keys,
BTreeMap::from([
("a.b\"c".to_string(), operand::EXISTS),
("author".to_string(), operand::STRING),
("f".to_string(), operand::BOOL),
("n".to_string(), operand::NUMBER),
])
);
}
/// Counting walks the segments, keeps to the scope and stops once over
/// the limit.
#[tokio::test]
async fn counts_follow_segments_and_scope() {
let dir = TempDir::new().unwrap();
let mut dataset = dataset(&dir, 1000).await;
dataset.drop_index("_mk_0_idx").await.unwrap();
// Ten fragments of 100 rows: four segments of up to 300 rows.
let ids: Vec<u64> = dataset.fragments().iter().map(|f| f.id).collect();
metadata_index::index_fragments(&mut dataset, "_mk_0", 300, 0)
.await
.unwrap();
let segments = metadata_index::segments(&dataset, "_mk_0").await.unwrap();
assert_eq!(segments.len(), 4);
let ready = ready_columns(&dataset);
let indexed =
crate::filter::apply(&mut dataset.scan(), &eq("author", "\"rare\""), &ready).unwrap();
let rare = &indexed[0];
// "rare" in rows i % 10 == 0, "rare2" never matches it.
assert_eq!(
count_indexed(&dataset, rare, None, 1_000).await.unwrap(),
Some(100)
);
assert_eq!(count_indexed(&dataset, rare, None, 99).await.unwrap(), None);
assert_eq!(
count_indexed(&dataset, rare, Some(&ids[..2]), 1_000)
.await
.unwrap(),
Some(20)
);
assert_eq!(
count_indexed(&dataset, rare, Some(&ids[9..]), 1_000)
.await
.unwrap(),
Some(10)
);
assert_eq!(
count_indexed(&dataset, rare, Some(&[]), 1_000)
.await
.unwrap(),
Some(0)
);
}
}
+1
View File
@@ -8,6 +8,7 @@
pub mod coverage;
mod filter;
pub mod index;
pub mod indexed_filter;
pub mod maintenance;
pub mod query;
pub mod segments;
+1
View File
@@ -309,6 +309,7 @@ async fn main() -> Result<()> {
nprobes,
index_segments: None,
fragments: None,
exact_match_limit: None,
};
let hits = if flat {
query::flat(&dataset, &vector_table, &query).await?
@@ -312,6 +312,7 @@ mod tests {
nprobes: None,
index_segments: None,
fragments: None,
exact_match_limit: None,
};
let mut ids: Vec<u64> = query::flat(&pinned, &resolved, &q)
.await
@@ -734,6 +735,7 @@ mod tests {
nprobes: None,
index_segments: None,
fragments: None,
exact_match_limit: None,
};
assert!(query::flat(&dataset, &resolved, &q).await.is_err());
}
+143 -6
View File
@@ -39,8 +39,15 @@ pub struct Query {
/// a worker the unindexed-fragment sweep. Listed ids that no longer exist
/// scan nothing.
pub fragments: Option<Vec<u64>>,
/// Score the matches exactly instead of searching the vector index when
/// the filter's indexed metadata keys match at most this many rows of the
/// scope; 0 never does. None takes [`DEFAULT_EXACT_MATCH_LIMIT`].
pub exact_match_limit: Option<usize>,
}
/// Up to this many matches of an indexed metadata key are scored exactly.
pub const DEFAULT_EXACT_MATCH_LIMIT: usize = 256;
/// One row of a top-K answer.
#[derive(Debug, Clone, serde::Serialize)]
pub struct Hit {
@@ -53,14 +60,101 @@ pub struct Hit {
/// Exact top-K over every fragment, ignoring any index.
pub async fn flat(dataset: &Dataset, table: &VectorTable, query: &Query) -> Result<Vec<Hit>> {
run(scanner(dataset, table, query, false).await?, table).await
run(scanner(dataset, table, query, false).await?.0, table).await
}
/// Top-K with the ANN index where it covers the fragment and a flat pass over
/// the rest. Whether the index is used is Lance's decision per fragment, so
/// rows written after the last index build are always candidates.
///
/// A prefiltered search sees only the probed partitions: it can come back
/// short of k, or with unscored rows (infinite distance) when at most k rows
/// match. Either way it is answered again by an exact scan of the same scope.
///
/// When the filter's indexed metadata keys match few rows of that scope,
/// they are scored exactly to begin with. The decision is per scope, so
/// exact answers of several scopes merge into the exact answer.
pub async fn ann(dataset: &Dataset, table: &VectorTable, query: &Query) -> Result<Vec<Hit>> {
run(scanner(dataset, table, query, true).await?, table).await
if few_indexed_matches(dataset, table, query).await? {
return exact_in_scope(dataset, table, query).await;
}
let (scanner, indexed) = scanner(dataset, table, query, true).await?;
let hits = run(scanner, table).await?;
if indexed
&& query.filter.is_some()
&& !query.postfilter
&& (hits.len() < query.k || hits.iter().any(|h| !h.distance.is_finite()))
{
return exact_in_scope(dataset, table, query).await;
}
Ok(hits)
}
/// [`flat`] over what [`ann`] searches for `query`: the listed fragments plus
/// the live coverage of the listed index segments, or the whole table.
async fn exact_in_scope(dataset: &Dataset, table: &VectorTable, query: &Query) -> Result<Vec<Hit>> {
let exact = Query {
index_segments: None,
fragments: scope(dataset, table, query).await?,
..query.clone()
};
run(scanner(dataset, table, &exact, false).await?.0, table).await
}
/// The fragments [`ann`] searches for `query`, or None for the whole table.
async fn scope(dataset: &Dataset, table: &VectorTable, query: &Query) -> Result<Option<Vec<u64>>> {
Ok(match &query.index_segments {
Some(segments) if !segments.is_empty() => {
let mut ids = query.fragments.clone().unwrap_or_default();
ids.extend(
crate::segments::covered_fragments(dataset, &table.vector_column, segments).await?,
);
Some(ids)
}
_ => query.fragments.clone(),
})
}
/// Whether the rows of [`ann`]'s scope matching the filter's indexed
/// metadata keys, counted through their scalar indexes, are at most the
/// query's limit. Only a search through the vector index asks.
pub(crate) async fn few_indexed_matches(
dataset: &Dataset,
table: &VectorTable,
query: &Query,
) -> Result<bool> {
let limit = query.exact_match_limit.unwrap_or(DEFAULT_EXACT_MATCH_LIMIT);
let Some(filter) = &query.filter else {
return Ok(false);
};
if limit == 0 || query.postfilter || query.index_segments.as_ref().is_some_and(|s| s.is_empty())
{
return Ok(false);
}
let ready = crate::indexed_filter::ready_columns(dataset);
if ready.is_empty()
|| crate::index::index_names(dataset, &table.vector_column)
.await?
.is_empty()
{
return Ok(false);
}
let indexed = crate::filter::apply(&mut dataset.scan(), filter, &ready)?;
if indexed.is_empty() {
return Ok(false);
}
let scope = scope(dataset, table, query).await?;
// An AND matches at most what its most selective indexed conjunct does.
for conjunct in &indexed {
let count =
crate::indexed_filter::count_indexed(dataset, conjunct, scope.as_deref(), limit as u64)
.await?;
tracing::debug!(column = %conjunct.column, ?count, limit, "indexed metadata filter");
if count.is_some() {
return Ok(true);
}
}
Ok(false)
}
/// Fetch rows by key, projected to `columns` (all of them when None).
@@ -101,7 +195,7 @@ async fn scanner(
table: &VectorTable,
query: &Query,
use_index: bool,
) -> Result<Scanner> {
) -> Result<(Scanner, bool)> {
if query.vector.len() != table.dimension {
bail!(
"query dim({}) does not match the table's dim({})",
@@ -117,18 +211,24 @@ async fn scanner(
.use_index(use_index)
.with_row_id();
if let Some(filter) = &query.filter {
crate::filter::apply(&mut scanner, filter)?;
let ready = crate::indexed_filter::ready_columns(dataset);
crate::filter::apply(&mut scanner, filter, &ready)?;
scanner.prefilter(!query.postfilter);
}
// An explicitly empty segment list searches no index, so there is
// nothing to size probes for.
let no_segments = query.index_segments.as_ref().is_some_and(|s| s.is_empty());
let mut indexed = false;
if use_index && !no_segments {
let probes = match query.nprobes {
Some(n) => Some(crate::index::Probes { min: n, max: n }),
Some(n) => crate::index::index_names(dataset, &table.vector_column)
.await?
.first()
.map(|_| crate::index::Probes { min: n, max: n }),
None => crate::index::default_probes(dataset, &table.vector_column).await?,
};
if let Some(probes) = probes {
indexed = true;
// Lance searches `min` partitions, and under a prefilter keeps
// going up to `max` until it has k rows. An open bound would
// make it prepare every partition for that, which costs as much
@@ -178,7 +278,7 @@ async fn scanner(
columns.push(DISTANCE_COLUMN.to_string());
columns.push(ROW_ID_COLUMN.to_string());
scanner.project(&columns)?;
Ok(scanner)
Ok((scanner, indexed))
}
async fn run(scanner: Scanner, table: &VectorTable) -> Result<Vec<Hit>> {
@@ -261,6 +361,7 @@ mod tests {
nprobes: None,
index_segments: None,
fragments: None,
exact_match_limit: None,
};
let hits = flat(&dataset, &table, &query).await.unwrap();
assert_eq!(hits.len(), 2);
@@ -302,6 +403,7 @@ mod tests {
nprobes: None,
index_segments: None,
fragments: None,
exact_match_limit: None,
},
)
.await
@@ -381,6 +483,7 @@ mod tests {
nprobes: None,
index_segments: None,
fragments: None,
exact_match_limit: None,
},
)
.await
@@ -484,6 +587,7 @@ mod tests {
nprobes: None,
index_segments: None,
fragments: None,
exact_match_limit: None,
};
assert!(flat(&dataset, &table, &query).await.is_err());
}
@@ -507,6 +611,7 @@ mod tests {
nprobes: None,
index_segments: None,
fragments: None,
exact_match_limit: None,
};
let indexed = ann(&dataset, &table, &query).await.unwrap();
let exact = flat(&dataset, &table, &query).await.unwrap();
@@ -515,4 +620,36 @@ mod tests {
exact.iter().map(|h| &h.key).collect::<Vec<_>>()
);
}
/// An explicit probe count does not make an unindexed table's scan count
/// as indexed, so a short filtered answer is not scanned a second time.
#[tokio::test]
async fn explicit_probes_need_an_index() {
let dir = TempDir::new().unwrap();
let rows = crate::testing::vectors(300, DIM);
let mut dataset = write_dataset(&dir, &rows, &["a"; 300]).await;
let table = VectorTable::resolve(&dataset, None, None).unwrap();
let query = Query {
vector: rows[0].clone(),
k: 10,
filter: Some("tag = 'a'".to_string()),
postfilter: false,
distance: DistanceType::L2,
nprobes: Some(4),
index_segments: None,
fragments: None,
exact_match_limit: None,
};
assert!(!scanner(&dataset, &table, &query, true).await.unwrap().1);
crate::index::build(
&mut dataset,
&table.vector_column,
crate::index::INDEX_NAME,
4,
false,
)
.await
.unwrap();
assert!(scanner(&dataset, &table, &query, true).await.unwrap().1);
}
}
@@ -139,6 +139,7 @@ mod tests {
nprobes: None,
index_segments: None,
fragments: None,
exact_match_limit: None,
};
let want: Vec<(u64, u32)> = query::ann(&dataset, &resolved, &base)
.await
@@ -239,6 +240,7 @@ mod tests {
nprobes: None,
index_segments: None,
fragments: Some(vec![u64::MAX]),
exact_match_limit: None,
};
assert!(
query::ann(&dataset, &resolved, &q)
+1
View File
@@ -44,6 +44,7 @@ fn query_for(vector: Vec<f32>, k: usize) -> Query {
nprobes: Some(4),
index_segments: None,
fragments: None,
exact_match_limit: None,
}
}
+1
View File
@@ -46,6 +46,7 @@ tracing.workspace = true
tracing-subscriber.workspace = true
[dev-dependencies]
lance-index = "10"
tempfile = "3"
[build-dependencies]
@@ -0,0 +1,392 @@
//! How often, and how slowly, each index's queries filter on each metadata
//! key over the last hour. Keys come from the parsed filter; the number of
//! indexes and of keys per index is bounded, so arbitrary keys cannot grow
//! memory without limit. Not exported as metrics: a key is no label.
use std::collections::{BTreeMap, HashMap};
use std::sync::Mutex;
use std::time::{Duration, Instant};
use weed_vector_core::indexed_filter::operand;
use crate::pb::vector as pb;
const BUCKETS: usize = 12;
const BUCKET: Duration = Duration::from_secs(300);
pub const WINDOW: Duration = Duration::from_secs(BUCKET.as_secs() * BUCKETS as u64);
pub const MAX_INDEXES: usize = 512;
pub const MAX_KEYS_PER_INDEX: usize = 32;
const MAX_RETIRED_TABLES: usize = 4;
#[derive(Default, Clone, Copy)]
struct Bucket {
/// The bucket's number since the stats started.
at: u64,
queries: u64,
seconds: f64,
/// The operand kinds the bucket's queries compared the key with.
kinds: u8,
}
#[derive(Default)]
struct KeyStats {
buckets: [Bucket; BUCKETS],
}
#[derive(Default)]
struct IndexStats {
/// The table the stats were counted on: an index dropped and created
/// again under its name is another table.
table: String,
/// Tables the name held before, whose late queries are not counted.
retired: Vec<String>,
keys: HashMap<String, KeyStats>,
/// The last bucket a key was left uncounted in.
truncated: Option<u64>,
last: u64,
}
impl IndexStats {
/// Starts counting on `table`, retiring the table counted on before.
fn replace(&mut self, table: &str) {
let mut retired = std::mem::take(&mut self.retired);
if !self.table.is_empty() {
retired.push(std::mem::take(&mut self.table));
if retired.len() > MAX_RETIRED_TABLES {
retired.remove(0);
}
}
*self = IndexStats {
table: table.to_string(),
retired,
last: self.last,
..Default::default()
};
}
}
/// Whether bucket `at` is still in the window of bucket `now`.
fn live(at: u64, now: u64) -> bool {
at + BUCKETS as u64 > now
}
pub struct FilterStats {
start: Instant,
indexes: Mutex<HashMap<String, IndexStats>>,
}
impl Default for FilterStats {
fn default() -> Self {
Self::new()
}
}
impl FilterStats {
pub fn new() -> Self {
Self {
start: Instant::now(),
indexes: Mutex::new(HashMap::new()),
}
}
fn bucket(&self, now: Instant) -> u64 {
(now.duration_since(self.start).as_secs() / BUCKET.as_secs()) + 1
}
/// Drops the stats of `index`, which now holds `table` ("" once
/// dropped): late queries of the table it held before are not counted.
pub fn forget(&self, index: &str, table: &str) {
let mut indexes = self.indexes.lock().unwrap();
if let Some(stats) = indexes.get_mut(index)
&& (table.is_empty() || stats.table != table)
{
stats.replace(table);
}
}
/// One query of `index`, whose table is `table`, that took `elapsed` and
/// filtered on `keys`. Stats counted on another table under the name are
/// dropped; a query of a table the name no longer holds is not counted.
pub fn record(&self, index: &str, table: &str, keys: &BTreeMap<String, u8>, elapsed: Duration) {
self.record_table_at(index, table, keys, elapsed, Instant::now());
}
#[cfg(test)]
fn record_at(&self, index: &str, keys: &BTreeMap<String, u8>, elapsed: Duration, now: Instant) {
self.record_table_at(index, "", keys, elapsed, now);
}
fn record_table_at(
&self,
index: &str,
table: &str,
keys: &BTreeMap<String, u8>,
elapsed: Duration,
now: Instant,
) {
if keys.is_empty() {
return;
}
let at = self.bucket(now);
let mut indexes = self.indexes.lock().unwrap();
if !indexes.contains_key(index) && indexes.len() >= MAX_INDEXES {
let Some(oldest) = indexes
.iter()
.min_by_key(|(_, stats)| stats.last)
.map(|(name, _)| name.clone())
else {
return;
};
indexes.remove(&oldest);
}
let stats = indexes.entry(index.to_string()).or_default();
if !table.is_empty() && stats.table != table {
if stats.retired.iter().any(|t| t == table) {
return;
}
if stats.table.is_empty() {
stats.table = table.to_string();
} else {
stats.replace(table);
}
}
stats.last = stats.last.max(at);
for (key, kinds) in keys {
if !stats.keys.contains_key(key) && stats.keys.len() >= MAX_KEYS_PER_INDEX {
// Make room only by dropping a key no query used in the window.
let idle = stats
.keys
.iter()
.find(|(_, k)| k.buckets.iter().all(|b| !live(b.at, at)))
.map(|(name, _)| name.clone());
match idle {
Some(idle) => {
stats.keys.remove(&idle);
}
None => {
stats.truncated = stats.truncated.max(Some(at));
continue;
}
}
}
let entry = stats.keys.entry(key.clone()).or_default();
let bucket = &mut entry.buckets[(at % BUCKETS as u64) as usize];
if bucket.at != at {
*bucket = Bucket {
at,
..Default::default()
};
}
bucket.queries += 1;
bucket.seconds += elapsed.as_secs_f64();
bucket.kinds |= kinds;
}
}
/// The stats of `index` over the window.
pub fn snapshot(&self, index: &str) -> pb::FilterStatsResponse {
self.snapshot_at(index, Instant::now())
}
fn snapshot_at(&self, index: &str, now: Instant) -> pb::FilterStatsResponse {
let at = self.bucket(now);
let window_seconds = now.duration_since(self.start).min(WINDOW).as_secs() as u32;
let indexes = self.indexes.lock().unwrap();
let Some(stats) = indexes.get(index) else {
return pb::FilterStatsResponse {
window_seconds,
..Default::default()
};
};
let mut keys: Vec<pb::FilterKeyStats> = stats
.keys
.iter()
.filter_map(|(key, s)| {
let (queries, seconds, kinds) = s
.buckets
.iter()
.filter(|b| live(b.at, at))
.fold((0, 0.0, 0), |(q, t, k), b| {
(q + b.queries, t + b.seconds, k | b.kinds)
});
(queries > 0).then(|| pb::FilterKeyStats {
key: key.clone(),
queries,
seconds,
string_operand: kinds & operand::STRING != 0,
number_operand: kinds & operand::NUMBER != 0,
bool_operand: kinds & operand::BOOL != 0,
})
})
.collect();
keys.sort_by(|a, b| b.queries.cmp(&a.queries).then(a.key.cmp(&b.key)));
pb::FilterStatsResponse {
keys,
window_seconds,
truncated: stats.truncated.is_some_and(|t| live(t, at)),
}
}
}
#[cfg(test)]
mod tests {
use super::*;
fn keys(pairs: &[(&str, u8)]) -> BTreeMap<String, u8> {
pairs.iter().map(|(k, v)| (k.to_string(), *v)).collect()
}
#[test]
fn counts_queries_and_latency_per_key_within_the_window() {
let stats = FilterStats::new();
let t0 = stats.start;
let ms = Duration::from_millis;
stats.record_at("b/i", &keys(&[("author", operand::STRING)]), ms(10), t0);
stats.record_at(
"b/i",
&keys(&[("author", operand::STRING), ("year", operand::NUMBER)]),
ms(30),
t0 + Duration::from_secs(600),
);
stats.record_at("b/j", &keys(&[("x", operand::BOOL)]), ms(5), t0);
let got = stats.snapshot_at("b/i", t0 + Duration::from_secs(900));
assert_eq!(got.window_seconds, 900);
assert_eq!(got.keys.len(), 2);
assert_eq!(got.keys[0].key, "author");
assert_eq!(got.keys[0].queries, 2);
assert!((got.keys[0].seconds - 0.04).abs() < 1e-9);
assert!(got.keys[0].string_operand && !got.keys[0].number_operand);
assert_eq!(got.keys[1].key, "year");
assert!(got.keys[1].number_operand);
// The first query leaves the window an hour after its bucket.
let later = stats.snapshot_at("b/i", t0 + WINDOW + Duration::from_secs(1));
assert_eq!(later.window_seconds, WINDOW.as_secs() as u32);
assert_eq!(later.keys[0].queries, 1);
let year = later.keys.iter().find(|k| k.key == "year").unwrap();
assert!(year.number_operand);
// A string query that left the window no longer marks the key.
stats.record_at(
"b/i",
&keys(&[("author", operand::NUMBER)]),
ms(1),
t0 + WINDOW + Duration::from_secs(2),
);
let mixed = stats.snapshot_at("b/i", t0 + WINDOW + Duration::from_secs(3));
let author = mixed.keys.iter().find(|k| k.key == "author").unwrap();
assert!(author.string_operand && author.number_operand);
let numeric = stats.snapshot_at("b/i", t0 + WINDOW + Duration::from_secs(700));
let author = numeric.keys.iter().find(|k| k.key == "author").unwrap();
assert!(!author.string_operand && author.number_operand);
let gone = stats.snapshot_at("b/i", t0 + 2 * WINDOW + Duration::from_secs(900));
assert!(gone.keys.is_empty());
assert!(stats.snapshot_at("other", t0).keys.is_empty());
}
#[test]
fn indexes_and_keys_are_bounded() {
let stats = FilterStats::new();
let t0 = stats.start;
for i in 0..MAX_KEYS_PER_INDEX + 5 {
stats.record_at(
"b/i",
&keys(&[(&format!("k{i}"), operand::STRING)]),
Duration::ZERO,
t0,
);
}
let got = stats.snapshot_at("b/i", t0);
assert_eq!(got.keys.len(), MAX_KEYS_PER_INDEX);
assert!(got.truncated);
// A key idle for the whole window makes room for a new one.
let later = t0 + WINDOW + BUCKET;
stats.record_at(
"b/i",
&keys(&[("new", operand::STRING)]),
Duration::ZERO,
later,
);
let got = stats.snapshot_at("b/i", later);
assert!(!got.truncated, "the uncounted keys left the window");
assert_eq!(got.keys.len(), 1);
assert_eq!(got.keys[0].key, "new");
for i in 0..MAX_INDEXES + 3 {
stats.record_at(
&format!("b/{i}"),
&keys(&[("k", operand::STRING)]),
Duration::ZERO,
later,
);
}
assert_eq!(stats.indexes.lock().unwrap().len(), MAX_INDEXES);
}
#[test]
fn a_record_from_an_earlier_bucket_keeps_the_later_truncation() {
let stats = FilterStats::new();
let t0 = stats.start;
for i in 0..=MAX_KEYS_PER_INDEX {
stats.record_at(
"b/i",
&keys(&[(&format!("k{i}"), operand::STRING)]),
Duration::ZERO,
t0 + BUCKET,
);
}
// Took its time before the lock while the later bucket recorded.
stats.record_at(
"b/i",
&keys(&[("late", operand::STRING)]),
Duration::ZERO,
t0,
);
let got = stats.snapshot_at("b/i", t0 + WINDOW);
assert!(
got.truncated,
"the later bucket's uncounted key is in the window"
);
assert_eq!(stats.indexes.lock().unwrap()["b/i"].last, 2);
}
#[test]
fn stats_follow_the_table_behind_an_index_name() {
let stats = FilterStats::new();
let t0 = stats.start;
let author = keys(&[("author", operand::STRING)]);
let count = |stats: &FilterStats| {
stats
.snapshot_at("b/i", t0)
.keys
.first()
.map_or(0, |k| k.queries)
};
stats.record_table_at("b/i", "t1", &author, Duration::ZERO, t0);
stats.record_table_at("b/i", "t1", &author, Duration::ZERO, t0);
assert_eq!(count(&stats), 2);
// The index was created again: its first query starts over.
stats.record_table_at("b/i", "t2", &author, Duration::ZERO, t0);
assert_eq!(count(&stats), 1);
// A late query of the old table is not counted on the new one.
stats.record_table_at("b/i", "t1", &author, Duration::ZERO, t0);
assert_eq!(count(&stats), 1);
stats.forget("b/i", "");
assert_eq!(count(&stats), 0);
// A first query records its table: a later table starts over.
stats.record_table_at("b/i", "t3", &author, Duration::ZERO, t0);
stats.record_table_at("b/i", "t4", &author, Duration::ZERO, t0);
assert_eq!(count(&stats), 1);
}
#[test]
fn a_late_query_after_a_drop_is_not_counted_on_the_new_table() {
let stats = FilterStats::new();
let t0 = stats.start;
let author = keys(&[("author", operand::STRING)]);
stats.record_table_at("b/i", "t1", &author, Duration::ZERO, t0);
stats.forget("b/i", "");
stats.record_table_at("b/i", "t2", &author, Duration::ZERO, t0);
stats.record_table_at("b/i", "t2", &author, Duration::ZERO, t0);
stats.record_table_at("b/i", "t1", &author, Duration::ZERO, t0);
assert_eq!(stats.snapshot_at("b/i", t0).keys[0].queries, 2);
}
}
@@ -0,0 +1,817 @@
//! Indexed metadata keys through the VectorService: PutVectors derives the
//! columns, and a query answers the same with or without them.
use std::sync::Arc;
use arrow_array::RecordBatch;
use arrow_schema::Schema;
use lance::Dataset;
use lance::dataset::{BatchUDF, NewColumnTransform};
use lance::index::DatasetIndexExt;
use lance_index::IndexType;
use lance_index::scalar::{BuiltinIndexType, ScalarIndexParams};
use tonic::Request;
use weed_lance_worker::metadata_index::{self, STATE_FIELD_META, State};
use weed_vector_core::query::{self, Query};
use weed_vector_core::table::VectorTable;
use crate::pb::vector::{self as pb, vector_service_server::VectorService};
use crate::service::VectorServer;
const DIM: usize = 8;
const ROWS: usize = 3000;
const COLUMN: &str = "_mk_0";
/// How the engine answers a filter once `author` is indexed.
#[derive(Debug, Clone, Copy, PartialEq)]
enum Path {
/// Through the column, at most the exact-match limit of rows: exact.
Exact,
/// Through the column, more rows: the vector index with the column's
/// prefilter.
Ivf,
/// Not through the column.
Json,
}
/// S3 Vectors filters, the gateway's translation of each, and its path.
const FILTERS: [(&str, &str, Path); 22] = [
(
"{\"author\":\"alice\"}",
"(COALESCE(json_extract(`metadata`, '$[\"author\"]') = '\"alice\"', false) OR COALESCE(json_array_contains(`metadata`, '$[\"author\"]', '\"alice\"'), false))",
Path::Ivf,
),
(
"{\"author\":{\"$eq\":\"bob\"}}",
"(COALESCE(json_extract(`metadata`, '$[\"author\"]') = '\"bob\"', false) OR COALESCE(json_array_contains(`metadata`, '$[\"author\"]', '\"bob\"'), false))",
Path::Exact,
),
(
"{\"author\":\"carol\"}",
"(COALESCE(json_extract(`metadata`, '$[\"author\"]') = '\"carol\"', false) OR COALESCE(json_array_contains(`metadata`, '$[\"author\"]', '\"carol\"'), false))",
Path::Exact,
),
(
"{\"author\":\"x\"}",
"(COALESCE(json_extract(`metadata`, '$[\"author\"]') = '\"x\"', false) OR COALESCE(json_array_contains(`metadata`, '$[\"author\"]', '\"x\"'), false))",
Path::Exact,
),
(
"{\"author\":\"7\"}",
"(COALESCE(json_extract(`metadata`, '$[\"author\"]') = '\"7\"', false) OR COALESCE(json_array_contains(`metadata`, '$[\"author\"]', '\"7\"'), false))",
Path::Exact,
),
(
"{\"author\":7}",
"(COALESCE(TRY_CAST(json_extract(`metadata`, '$[\"author\"]') AS DOUBLE) = 7, false) OR COALESCE(json_array_contains(`metadata`, '$[\"author\"]', '7'), false))",
Path::Json,
),
(
"{\"author\":true}",
"(COALESCE(json_extract(`metadata`, '$[\"author\"]') = 'true', false) OR COALESCE(json_array_contains(`metadata`, '$[\"author\"]', 'true'), false))",
Path::Json,
),
(
"{\"author\":{\"$in\":[\"alice\",\"bob\",\"x\"]}}",
"(COALESCE(json_extract(`metadata`, '$[\"author\"]') in ('\"alice\"', '\"bob\"', '\"x\"'), false) OR COALESCE(json_array_contains(`metadata`, '$[\"author\"]', '\"alice\"'), false) OR COALESCE(json_array_contains(`metadata`, '$[\"author\"]', '\"bob\"'), false) OR COALESCE(json_array_contains(`metadata`, '$[\"author\"]', '\"x\"'), false))",
Path::Ivf,
),
(
"{\"author\":{\"$in\":[\"alice\",7]}}",
"(COALESCE(json_extract(`metadata`, '$[\"author\"]') in ('\"alice\"'), false) OR COALESCE(TRY_CAST(json_extract(`metadata`, '$[\"author\"]') AS DOUBLE) in (7), false) OR COALESCE(json_array_contains(`metadata`, '$[\"author\"]', '\"alice\"'), false) OR COALESCE(json_array_contains(`metadata`, '$[\"author\"]', '7'), false))",
Path::Json,
),
(
"{\"author\":{\"$ne\":\"alice\"}}",
"NOT (COALESCE(json_extract(`metadata`, '$[\"author\"]') = '\"alice\"', false) OR COALESCE(json_array_contains(`metadata`, '$[\"author\"]', '\"alice\"'), false))",
Path::Json,
),
(
"{\"author\":{\"$nin\":[\"alice\",\"bob\"]}}",
"NOT (COALESCE(json_extract(`metadata`, '$[\"author\"]') in ('\"alice\"', '\"bob\"'), false) OR COALESCE(json_array_contains(`metadata`, '$[\"author\"]', '\"alice\"'), false) OR COALESCE(json_array_contains(`metadata`, '$[\"author\"]', '\"bob\"'), false))",
Path::Json,
),
(
"{\"author\":{\"$exists\":false}}",
"NOT json_exists(`metadata`, '$[\"author\"]')",
Path::Json,
),
(
"{\"$and\":[{\"author\":\"alice\"},{\"genre\":\"g1\"}]}",
"((COALESCE(json_extract(`metadata`, '$[\"author\"]') = '\"alice\"', false) OR COALESCE(json_array_contains(`metadata`, '$[\"author\"]', '\"alice\"'), false)) AND (COALESCE(json_extract(`metadata`, '$[\"genre\"]') = '\"g1\"', false) OR COALESCE(json_array_contains(`metadata`, '$[\"genre\"]', '\"g1\"'), false)))",
Path::Ivf,
),
(
"{\"$and\":[{\"author\":\"bob\"},{\"year\":{\"$gt\":2005}}]}",
"((COALESCE(json_extract(`metadata`, '$[\"author\"]') = '\"bob\"', false) OR COALESCE(json_array_contains(`metadata`, '$[\"author\"]', '\"bob\"'), false)) AND TRY_CAST(json_extract(`metadata`, '$[\"year\"]') AS DOUBLE) > 2005)",
Path::Exact,
),
(
"{\"$or\":[{\"author\":\"bob\"},{\"genre\":\"g2\"}]}",
"((COALESCE(json_extract(`metadata`, '$[\"author\"]') = '\"bob\"', false) OR COALESCE(json_array_contains(`metadata`, '$[\"author\"]', '\"bob\"'), false)) OR (COALESCE(json_extract(`metadata`, '$[\"genre\"]') = '\"g2\"', false) OR COALESCE(json_array_contains(`metadata`, '$[\"genre\"]', '\"g2\"'), false)))",
Path::Json,
),
(
"{\"author\":\"o'neil \\\"q\\\" \\\\ \\n\"}",
"(COALESCE(json_extract(`metadata`, '$[\"author\"]') = '\"o''neil \\\"q\\\" \\\\ \\n\"', false) OR COALESCE(json_array_contains(`metadata`, '$[\"author\"]', '\"o''neil \\\"q\\\" \\\\ \\n\"'), false))",
Path::Exact,
),
(
"{\"author\":\"nobody\"}",
"(COALESCE(json_extract(`metadata`, '$[\"author\"]') = '\"nobody\"', false) OR COALESCE(json_array_contains(`metadata`, '$[\"author\"]', '\"nobody\"'), false))",
Path::Exact,
),
("{\"author\":{\"$in\":[]}}", "false", Path::Json),
(
"{\"genre\":\"g1\"}",
"(COALESCE(json_extract(`metadata`, '$[\"genre\"]') = '\"g1\"', false) OR COALESCE(json_array_contains(`metadata`, '$[\"genre\"]', '\"g1\"'), false))",
Path::Json,
),
(
"{\"author\":\"u5\"}",
"(COALESCE(json_extract(`metadata`, '$[\"author\"]') = '\"u5\"', false) OR COALESCE(json_array_contains(`metadata`, '$[\"author\"]', '\"u5\"'), false))",
Path::Exact,
),
(
"{\"author\":\"ctl\\u0001x\"}",
"(COALESCE(json_extract(`metadata`, '$[\"author\"]') = '\"ctl\\u{1}x\"', false) OR COALESCE(json_array_contains(`metadata`, '$[\"author\"]', '\"ctl\\u0001x\"'), false))",
Path::Json,
),
(
"{\"author\":{\"$in\":[\"u5\",\"u6\"]}}",
"(COALESCE(json_extract(`metadata`, '$[\"author\"]') in ('\"u5\"', '\"u6\"'), false) OR COALESCE(json_array_contains(`metadata`, '$[\"author\"]', '\"u5\"'), false) OR COALESCE(json_array_contains(`metadata`, '$[\"author\"]', '\"u6\"'), false))",
Path::Exact,
),
];
fn rng(seed: &mut u64) -> f32 {
*seed = seed
.wrapping_mul(6364136223846793005)
.wrapping_add(1442695040888963407);
((*seed >> 33) as f32 / (1u64 << 31) as f32) - 0.5
}
fn vector_of(i: usize) -> Vec<f32> {
let mut centre = (i % 16) as u64 + 1;
let mut noise = i as u64 * 7919 + 13;
(0..DIM)
.map(|_| rng(&mut centre) * 4.0 + rng(&mut noise))
.collect()
}
fn author(i: usize) -> Option<serde_json::Value> {
use serde_json::json;
Some(match i % 100 {
0..=9 => json!("alice"),
10 => json!("bob"),
11 => json!(["carol", "x"]),
12 => json!(7),
13 => json!("7"),
14 => serde_json::Value::Null,
15 => return None,
16 => json!(true),
17 => json!("o'neil \"q\" \\ \n"),
18 => json!("ctl\u{1}x"),
19 => json!(["alice", 3]),
_ => json!(format!("u{}", i % 37)),
})
}
fn row(i: usize, author: Option<serde_json::Value>) -> pb::Vector {
let mut meta = serde_json::json!({"genre": format!("g{}", i % 3), "year": 2000 + i % 10});
if let Some(author) = author {
meta["author"] = author;
}
let serde_json::Value::Object(meta) = meta else {
unreachable!()
};
pb::Vector {
key: format!("k{i}"),
data: vector_of(i),
metadata: Some(crate::service::json_to_struct(meta)),
}
}
struct Fixture {
server: VectorServer,
exact_off: VectorServer,
table: pb::Table,
}
impl Fixture {
async fn new() -> Self {
let bucket = format!(
"mk-{:x}",
std::time::SystemTime::now()
.duration_since(std::time::UNIX_EPOCH)
.unwrap()
.as_nanos()
);
let table = pb::Table {
id: vec![bucket.clone(), "vector".into(), "x".into()],
location: format!("shared-memory://{bucket}/vector/x"),
..Default::default()
};
let fixture = Self {
server: VectorServer::new().with_schemes(&["shared-memory"]),
exact_off: VectorServer::new()
.with_schemes(&["shared-memory"])
.with_exact_match_limit(Some(0)),
table,
};
fixture
.server
.create_index(Request::new(pb::CreateIndexRequest {
table: Some(fixture.table.clone()),
dimension: DIM as u32,
distance_metric: pb::DistanceMetric::L2 as i32,
..Default::default()
}))
.await
.unwrap();
fixture
}
async fn put(&self, vectors: Vec<pb::Vector>) {
self.server
.put_vectors(Request::new(pb::PutVectorsRequest {
table: Some(self.table.clone()),
vectors,
}))
.await
.unwrap();
}
async fn dataset(&self, version: u64) -> Dataset {
let dataset = Dataset::open(&self.table.location).await.unwrap();
if version == 0 {
dataset
} else {
dataset.checkout_version(version).await.unwrap()
}
}
/// Adds `author`'s column, indexes it and marks it ready, returning the
/// version before it was ready.
async fn index_author(&self) -> u64 {
let mut dataset = self.dataset(0).await;
let output = Arc::new(Schema::new(vec![metadata_index::field(
COLUMN,
"author",
State::Building,
)]));
let schema = output.clone();
dataset
.add_columns(
NewColumnTransform::BatchUDF(BatchUDF {
mapper: Box::new(move |batch: &RecordBatch| {
let column = metadata_index::column_from_metadata(
batch.column_by_name("metadata").unwrap().as_ref(),
"author",
)
.map_err(|e| lance::Error::invalid_input(e.to_string()))?;
Ok(RecordBatch::try_new(schema.clone(), vec![column])?)
}),
output_schema: output,
result_checkpoint: None,
}),
Some(vec!["metadata".to_string()]),
None,
)
.await
.unwrap();
dataset
.create_index(
&[COLUMN],
IndexType::LabelList,
Some(format!("{COLUMN}_idx")),
&ScalarIndexParams::for_builtin(BuiltinIndexType::LabelList),
true,
)
.await
.unwrap();
let building = dataset.version().version;
self.set_state(State::Ready).await;
building
}
async fn set_state(&self, state: State) -> u64 {
let mut dataset = self.dataset(0).await;
dataset
.update_field_metadata()
.update(COLUMN, [(STATE_FIELD_META, state.as_str())])
.unwrap()
.await
.unwrap();
dataset.version().version
}
async fn query(
&self,
server: &VectorServer,
version: u64,
filter: &str,
k: u32,
v: &[f32],
) -> Vec<(String, f32)> {
let mut table = self.table.clone();
table.version = version;
server
.query_vectors(Request::new(pb::QueryVectorsRequest {
table: Some(table),
query_vector: v.to_vec(),
top_k: k,
filter: filter.to_string(),
return_distance: true,
..Default::default()
}))
.await
.unwrap()
.into_inner()
.vectors
.into_iter()
.map(|r| (r.key, r.distance))
.collect()
}
/// Exact filtered top-k at `version`.
async fn flat(&self, version: u64, filter: &str, k: u32, v: &[f32]) -> Vec<(String, f32)> {
let dataset = self.dataset(version).await;
let table = VectorTable::resolve(&dataset, None, Some("key")).unwrap();
query::flat(
&dataset,
&table,
&Query {
vector: v.to_vec(),
k: k as usize,
filter: Some(filter.to_string()),
postfilter: false,
distance: lance_linalg::distance::DistanceType::L2,
nprobes: None,
index_segments: None,
fragments: None,
exact_match_limit: None,
},
)
.await
.unwrap()
.into_iter()
.map(|h| (h.key, h.distance))
.collect()
}
/// Every filter answers through the column as through the JSON at
/// `json` (the same rows and indexes, the column not ready), and an
/// exact answer as the exact filtered top-k.
async fn compare(&self, json: u64, ready: u64) {
for q in [7, 1234] {
let v = vector_of(q);
for (s3, filter, path) in FILTERS {
for k in [10, 400] {
let what = format!("{s3} k={k} q={q}");
let exact = self.flat(json, filter, k, &v).await;
assert_same(&self.flat(ready, filter, k, &v).await, &exact, &what);
let json_ann = self.query(&self.server, json, filter, k, &v).await;
let off = self.query(&self.exact_off, ready, filter, k, &v).await;
assert_same(&off, &json_ann, &format!("{what}: indexed prefilter"));
let got = self.query(&self.server, ready, filter, k, &v).await;
let want = if path == Path::Exact {
&exact
} else {
&json_ann
};
assert_same(&got, want, &format!("{what}: {path:?}"));
}
}
}
}
}
fn assert_same(got: &[(String, f32)], want: &[(String, f32)], what: &str) {
assert_eq!(got.len(), want.len(), "{what}: {got:?} vs {want:?}");
// Ties may come back in either order: the same keys, and the same
// distance at each rank.
fn keys(hits: &[(String, f32)]) -> Vec<&str> {
let mut keys: Vec<&str> = hits.iter().map(|(k, _)| k.as_str()).collect();
keys.sort();
keys
}
assert_eq!(keys(got), keys(want), "{what}: {got:?} vs {want:?}");
for (g, w) in got.iter().zip(want) {
let same = (g.1 - w.1).abs() < 1e-4 || (g.1.is_infinite() && w.1.is_infinite());
assert!(same, "{what}: {got:?} vs {want:?}");
if g.0 != w.0 {
let tie = want
.iter()
.any(|(k, d)| k == &g.0 && (d - w.1).abs() < 1e-4);
assert!(
tie,
"{what}: {} at {} is no tie: {got:?} vs {want:?}",
g.0, g.1
);
}
}
}
/// Indexed and JSON answers agree on mixed types, missing keys, arrays,
/// null, untouched operators, and after updates, deletes and rows no
/// scalar or vector index covers yet.
#[tokio::test]
async fn indexed_answers_match_the_json_path() {
let fixture = Fixture::new().await;
for start in (0..ROWS).step_by(300) {
fixture
.put((start..start + 300).map(|i| row(i, author(i))).collect())
.await;
}
let mut dataset = fixture.dataset(0).await;
weed_lance_worker::vector_index::build(&mut dataset, "vector", "vector_idx", 16, false)
.await
.unwrap();
let json = fixture.index_author().await;
let schema: Schema = fixture.dataset(0).await.schema().into();
assert_eq!(
metadata_index::indexed_keys(&schema)[0].state,
Some(State::Ready)
);
fixture.compare(json, 0).await;
// Updates move rows between values, deletes drop some, and new rows land
// in fragments neither index covers; PutVectors fills the column.
fixture
.put(
(0..60)
.map(|i| row(i, Some(serde_json::json!(["moved", "bob"]))))
.chain((ROWS..ROWS + 200).map(|i| row(i, author(i))))
.collect(),
)
.await;
fixture
.server
.delete_vectors(Request::new(pb::DeleteVectorsRequest {
table: Some(fixture.table.clone()),
keys: (100..130).map(|i| format!("k{i}")).collect(),
}))
.await
.unwrap();
let json = fixture.set_state(State::Building).await;
let ready = fixture.set_state(State::Ready).await;
fixture.compare(json, ready).await;
let moved = r#"(COALESCE(json_extract(`metadata`, '$["author"]') = '"moved"', false) OR COALESCE(json_array_contains(`metadata`, '$["author"]', '"moved"'), false))"#;
let hits = fixture
.query(&fixture.server, ready, moved, 100, &vector_of(0))
.await;
assert_eq!(hits.len(), 60);
}
/// The planner counts and scores per scope: an exact answer over a fragment
/// subset is that subset's exact top-k.
#[tokio::test]
async fn a_scoped_query_is_exact_within_its_fragments() {
let fixture = Fixture::new().await;
for start in (0..ROWS).step_by(300) {
fixture
.put((start..start + 300).map(|i| row(i, author(i))).collect())
.await;
}
let mut dataset = fixture.dataset(0).await;
weed_lance_worker::vector_index::build(&mut dataset, "vector", "vector_idx", 16, false)
.await
.unwrap();
let json = fixture.index_author().await;
let ready = fixture.dataset(0).await;
let table = VectorTable::resolve(&ready, None, Some("key")).unwrap();
let ids: Vec<u64> = ready.fragments().iter().map(|f| f.id).take(4).collect();
let filter = FILTERS[1].1;
let scoped = || Query {
vector: vector_of(5),
k: 10,
filter: Some(filter.to_string()),
postfilter: false,
distance: lance_linalg::distance::DistanceType::L2,
nprobes: None,
index_segments: None,
fragments: Some(ids.clone()),
exact_match_limit: None,
};
let got = query::ann(&ready, &table, &scoped()).await.unwrap();
let old = fixture.dataset(json).await;
let want = query::flat(&old, &table, &scoped()).await.unwrap();
assert_eq!(want.len(), 10);
let pairs = |hits: Vec<weed_vector_core::Hit>| {
hits.into_iter()
.map(|h| (h.key, h.distance))
.collect::<Vec<_>>()
};
assert_same(&pairs(got), &pairs(want), "scoped bob");
}
/// The engine counts each index's filtered queries by key.
#[tokio::test]
async fn queries_are_counted_by_filter_key() {
let fixture = Fixture::new().await;
fixture
.put((0..50).map(|i| row(i, author(i))).collect())
.await;
for (_, filter, _) in &FILTERS[..3] {
fixture
.query(&fixture.server, 0, filter, 5, &vector_of(1))
.await;
}
let year = FILTERS[13].1;
fixture
.query(&fixture.server, 0, year, 5, &vector_of(1))
.await;
let stats = fixture
.server
.filter_stats(Request::new(pb::FilterStatsRequest {
table: Some(fixture.table.clone()),
}))
.await
.unwrap()
.into_inner();
let author = stats.keys.iter().find(|k| k.key == "author").unwrap();
assert_eq!(author.queries, 4);
assert!(author.string_operand && !author.number_operand);
let year = stats.keys.iter().find(|k| k.key == "year").unwrap();
assert_eq!(year.queries, 1);
assert!(year.number_operand && !year.string_operand);
}
/// A PutVectors that opened the table before its key column was added and
/// commits after: Lance rebases the write and its rows hold NULL in the new
/// column, which the write then fills from the rows' metadata.
#[tokio::test(flavor = "multi_thread", worker_threads = 4)]
async fn a_write_that_raced_the_column_add_fills_the_column() {
let fixture = Fixture::new().await;
fixture
.put((0..300).map(|i| row(i, author(i))).collect())
.await;
let unhealed = fixture.dataset(0).await;
let healed = fixture.dataset(0).await;
fixture.index_author().await;
let nulls = |dataset: Dataset| async move {
dataset
.count_rows(Some(format!("{COLUMN} IS NULL")))
.await
.unwrap()
};
let rows: Vec<pb::Vector> = (250..400).map(|i| row(i, author(i))).collect();
let written = crate::service::merge_vectors(unhealed, &rows)
.await
.unwrap();
assert!(written.schema().field(COLUMN).is_some());
assert!(nulls(written).await >= 100, "the race is reproduced");
let written = crate::service::write_vectors(healed, &rows).await.unwrap();
assert_eq!(nulls(written.clone()).await, 0);
let filter = FILTERS[1].1;
let latest = written.version().version;
let v = vector_of(5);
assert_same(
&fixture
.query(&fixture.server, latest, filter, 400, &v)
.await,
&fixture.flat(latest, filter, 400, &v).await,
"bob after the race",
);
assert_eq!(fixture.flat(latest, filter, 400, &v).await.len(), 4);
}
/// A write names no version: one at a version from before a key's column
/// add would commit rows without the column.
#[tokio::test(flavor = "multi_thread", worker_threads = 4)]
async fn a_write_at_an_older_version_is_refused() {
let fixture = Fixture::new().await;
fixture
.put((0..10).map(|i| row(i, author(i))).collect())
.await;
let before = fixture.dataset(0).await.version().version;
fixture.index_author().await;
let mut table = fixture.table.clone();
table.version = before;
let refused = fixture
.server
.put_vectors(Request::new(pb::PutVectorsRequest {
table: Some(table),
vectors: (10..20).map(|i| row(i, author(i))).collect(),
}))
.await
.unwrap_err();
assert_eq!(refused.code(), tonic::Code::InvalidArgument);
}
/// Two writes updating the same keys raced the column add, and the first
/// one's fill commits last: each write's rows get the values of its own
/// metadata, not the ones derived before it or from the other write.
#[tokio::test(flavor = "multi_thread", worker_threads = 4)]
async fn a_write_that_raced_the_column_add_derives_its_updated_rows() {
let fixture = Fixture::new().await;
fixture
.put((0..300).map(|i| row(i, author(i))).collect())
.await;
let first = fixture.dataset(0).await;
let second = fixture.dataset(0).await;
let written: Schema = first.schema().into();
fixture.index_author().await;
let rows = |name: &str| -> Vec<pb::Vector> {
(250..300)
.map(|i| row(i, Some(serde_json::json!(name))))
.collect()
};
let keys: Vec<String> = (250..300).map(|i| format!("k{i}")).collect();
let first = crate::service::merge_vectors(first, &rows("alice"))
.await
.unwrap();
let second = crate::service::merge_vectors(second, &rows("bob"))
.await
.unwrap();
crate::service::heal_columns(second, &written, &keys)
.await
.unwrap();
crate::service::heal_columns(first, &written, &keys)
.await
.unwrap();
let latest = fixture.dataset(0).await;
let count = |filter: String| {
let latest = latest.clone();
async move { latest.count_rows(Some(filter)).await.unwrap() }
};
assert_eq!(count(format!("{COLUMN} IS NULL")).await, 0);
for (name, filter) in [("alice", FILTERS[0].1), ("bob", FILTERS[1].1)] {
assert_eq!(
count(format!("array_has({COLUMN}, '{name}')")).await,
count(filter.to_string()).await,
"{name}"
);
}
let mut got = fixture
.server
.get_vectors(Request::new(pb::GetVectorsRequest {
table: Some(fixture.table.clone()),
keys: keys.clone(),
return_data: true,
return_metadata: true,
}))
.await
.unwrap()
.into_inner()
.vectors;
got.sort_by_key(|v| v.key[1..].parse::<usize>().unwrap());
assert_eq!(got, rows("bob"));
}
/// Operators ask through the engine; the table records the request and,
/// once the worker has done its part, the key's state, coverage and size.
#[tokio::test(flavor = "multi_thread", worker_threads = 4)]
async fn keys_are_requested_and_listed_through_the_engine() {
let fixture = Fixture::new().await;
fixture
.put((0..600).map(|i| row(i, author(i))).collect())
.await;
let update = |key: &str, remove: bool| {
fixture
.server
.update_metadata_key(Request::new(pb::UpdateMetadataKeyRequest {
table: Some(fixture.table.clone()),
key: key.to_string(),
remove,
}))
};
let listed = update("author", false).await.unwrap().into_inner();
assert_eq!(listed.keys.len(), 1);
assert_eq!(listed.keys[0].state, "requested");
assert_eq!(listed.keys[0].rows, 600);
// A retry of a committed request answers as the first one did.
let again = update("author", false).await.unwrap().into_inner();
assert_eq!(again.keys.len(), 1);
assert_eq!(again.keys[0].state, "requested");
let mut dataset = fixture.dataset(0).await;
let key = metadata_index::add_column(&mut dataset, "author")
.await
.unwrap();
metadata_index::finish_after(&mut dataset, &key, 1_000_000, std::time::Duration::ZERO)
.await
.unwrap();
let listed = fixture
.server
.metadata_keys(Request::new(pb::MetadataKeysRequest {
table: Some(fixture.table.clone()),
}))
.await
.unwrap()
.into_inner();
assert_eq!((listed.rows, listed.vector_index.as_str()), (600, ""));
let author = &listed.keys[0];
assert_eq!(
(
author.key.as_str(),
author.state.as_str(),
author.column.as_str()
),
("author", "ready", "_mk_0")
);
assert_eq!(
(author.indexed_rows, author.rows, author.segments),
(600, 600, 1)
);
assert!(author.index_bytes > 0);
// Off at once: the next query reads the JSON again.
let listed = update("author", true).await.unwrap().into_inner();
assert_eq!(listed.keys[0].state, "off");
let dataset = fixture.dataset(0).await;
assert_eq!(metadata_index::keys_of(&dataset)[0].state, Some(State::Off));
// Removing a key that is not indexed needs nothing.
let nobody = update("nobody", true).await.unwrap().into_inner();
assert_eq!(nobody.keys.len(), 1);
assert_eq!(
update("author", false).await.unwrap_err().code(),
tonic::Code::FailedPrecondition,
"being removed"
);
}
/// DeleteIndex purges the table's directory and an index created again
/// under the same name reuses it: the same running engine serves nothing of
/// the old table, neither rows, nor its indexed key, nor its filter stats.
#[tokio::test(flavor = "multi_thread", worker_threads = 4)]
async fn a_dropped_and_recreated_index_serves_nothing_old() {
let fixture = Fixture::new().await;
fixture
.put((0..600).map(|i| row(i, author(i))).collect())
.await;
let mut dataset = fixture.dataset(0).await;
weed_lance_worker::vector_index::build(&mut dataset, "vector", "vector_idx", 4, false)
.await
.unwrap();
fixture.index_author().await;
let bob = FILTERS[1].1;
let old = fixture
.query(&fixture.server, 0, bob, 50, &vector_of(10))
.await;
assert_eq!(old.len(), 6);
let server = &fixture.server;
let stats = || {
server.filter_stats(Request::new(pb::FilterStatsRequest {
table: Some(fixture.table.clone()),
}))
};
assert_eq!(stats().await.unwrap().into_inner().keys[0].queries, 1);
// What the catalog's DeleteTable does, then CreateIndex again.
let (store, path) = lance::io::ObjectStore::from_uri(&fixture.table.location)
.await
.unwrap();
store.remove_dir_all(path).await.unwrap();
fixture
.server
.create_index(Request::new(pb::CreateIndexRequest {
table: Some(fixture.table.clone()),
dimension: DIM as u32,
distance_metric: pb::DistanceMetric::L2 as i32,
..Default::default()
}))
.await
.unwrap();
let keys = fixture
.server
.metadata_keys(Request::new(pb::MetadataKeysRequest {
table: Some(fixture.table.clone()),
}))
.await
.unwrap()
.into_inner();
assert!(keys.keys.is_empty());
assert!(
stats().await.unwrap().into_inner().keys.is_empty(),
"the old table's filter stats"
);
fixture
.put((1000..1100).map(|i| row(i, author(i))).collect())
.await;
let new = |hits: &[(String, f32)]| {
hits.iter().all(|(k, _)| {
k[1..]
.parse::<usize>()
.is_ok_and(|i| (1000..1100).contains(&i))
})
};
let unfiltered = fixture
.query(&fixture.server, 0, "", 200, &vector_of(10))
.await;
assert_eq!(unfiltered.len(), 100);
assert!(new(&unfiltered), "{unfiltered:?}");
let bob_now = fixture
.query(&fixture.server, 0, bob, 50, &vector_of(10))
.await;
assert_eq!(bob_now.len(), 1);
assert!(new(&bob_now), "{bob_now:?}");
let author = stats().await.unwrap().into_inner().keys;
assert_eq!((author.len(), author[0].queries), (1, 1));
}
+7
View File
@@ -1,5 +1,12 @@
//! The VectorService gRPC server the S3 gateway talks to.
// PutVectors awaits two Lance merge_insert futures; proving the handler
// future Send takes deeper trait recursion than the default.
#![recursion_limit = "256"]
pub mod filter_stats;
#[cfg(test)]
mod indexed_keys_tests;
pub mod metrics;
pub mod observed;
pub mod read_cache;
+6
View File
@@ -52,6 +52,11 @@ struct Args {
/// tenth of the index's partitions, at least 8; more under a filter.
#[arg(long, env = "WEED_VECTOR_NPROBES")]
nprobes: Option<std::num::NonZeroUsize>,
/// A filtered query whose indexed metadata keys match at most this many
/// rows scores them exactly instead of searching the vector index; 0
/// never does.
#[arg(long, env = "WEED_VECTOR_EXACT_MATCH_LIMIT", default_value_t = weed_vector_core::query::DEFAULT_EXACT_MATCH_LIMIT)]
exact_match_limit: usize,
/// The cluster's security.toml: [grpc.vector] for this server,
/// [grpc.client] for the masters and filers, https.client for volume
/// servers under HTTPS, jwt.signing.read.key for signed volume reads,
@@ -170,6 +175,7 @@ async fn main() -> Result<()> {
.then(|| ReadCache::new(args.read_cache_mb << 20).with_metrics(metrics.clone()));
let mut service = VectorServer::new()
.with_nprobes(args.nprobes.map(std::num::NonZeroUsize::get))
.with_exact_match_limit(Some(args.exact_match_limit))
.with_lance_caches(args.index_cache_mb << 20, args.metadata_cache_mb << 20)
.with_storage(reads);
if let Some(cache) = &cache {
@@ -10,6 +10,10 @@ use tonic_health::{ServingStatus, server::HealthReporter};
use crate::read_cache::ReadCache;
/// The gRPC health service an engine reports serving when its writes derive
/// the indexed metadata key columns.
pub const METADATA_COLUMNS_CAPABILITY: &str = "weed.vector.capability.metadata_columns";
#[derive(Clone)]
pub struct Metrics {
pub registry: Registry,
@@ -120,6 +124,11 @@ impl Metrics {
health
.set_service_status("vector_pb.VectorService", status)
.await;
// A capability, not readiness: the admin refuses to index a
// metadata key while an engine without it could write the table.
health
.set_service_status(METADATA_COLUMNS_CAPABILITY, ServingStatus::Serving)
.await;
}
}
@@ -78,6 +78,33 @@ impl VectorService for ObservedService {
}
result
}
async fn metadata_keys(
&self,
request: Request<pb::MetadataKeysRequest>,
) -> Result<Response<pb::MetadataKeysResponse>, Status> {
let mut in_flight = self.metrics.start("MetadataKeys");
let result = self.inner.metadata_keys(request).await;
in_flight.complete(result.as_ref().err().map(|e| e.code()));
result
}
async fn update_metadata_key(
&self,
request: Request<pb::UpdateMetadataKeyRequest>,
) -> Result<Response<pb::MetadataKeysResponse>, Status> {
let mut in_flight = self.metrics.start("UpdateMetadataKey");
let result = self.inner.update_metadata_key(request).await;
in_flight.complete(result.as_ref().err().map(|e| e.code()));
result
}
async fn filter_stats(
&self,
request: Request<pb::FilterStatsRequest>,
) -> Result<Response<pb::FilterStatsResponse>, Status> {
let mut in_flight = self.metrics.start("FilterStats");
let result = self.inner.filter_stats(request).await;
in_flight.complete(result.as_ref().err().map(|e| e.code()));
result
}
}
#[cfg(test)]
+297 -48
View File
@@ -19,7 +19,9 @@ use weed_lance_worker::catalog::TableDescription;
use weed_lance_worker::dataset;
use weed_lance_worker::direct_read::{DirectReads, Wrappers};
use weed_lance_worker::jobs::vector_index::is_vector_table;
use weed_lance_worker::metadata_index;
use weed_vector_core::index;
use weed_vector_core::indexed_filter;
use weed_vector_core::maintenance::SharedLeases;
use weed_vector_core::query::{self, Query};
use weed_vector_core::table::VectorTable;
@@ -167,14 +169,13 @@ fn batch_of(vectors: &[pb::Vector], dimension: usize, schema: Arc<Schema>) -> Re
};
let vectors_array =
FixedSizeListArray::try_new(item_field.clone(), dimension as i32, values, None)?;
let metas = vectors
let json: Vec<Option<serde_json::Value>> = vectors
.iter()
.map(|v| {
v.metadata
.as_ref()
.map(|m| serde_json::to_string(&struct_to_json(m)))
.transpose()
})
.map(|v| v.metadata.as_ref().map(struct_to_json))
.collect();
let metas = json
.iter()
.map(|m| m.as_ref().map(serde_json::to_string).transpose())
.collect::<Result<Vec<_>, _>>()?;
// The stored column is lance's jsonb (LargeBinary) once the field carries
// the arrow.json extension; encode the strings, else keep the Utf8 batch.
@@ -187,11 +188,188 @@ fn batch_of(vectors: &[pb::Vector], dimension: usize, schema: Arc<Schema>) -> Re
}
_ => Arc::new(StringArray::from(metas)),
};
RecordBatch::try_new(
schema,
vec![Arc::new(keys), Arc::new(vectors_array), metadata_array],
let (keys, vectors_array): (ArrayRef, ArrayRef) = (Arc::new(keys), Arc::new(vectors_array));
// An indexed metadata key's column is derived from the row's metadata,
// whatever the key's state.
let columns = schema
.fields()
.iter()
.map(|field| match field.name().as_str() {
KEY_COLUMN => Ok(keys.clone()),
VECTOR_COLUMN => Ok(vectors_array.clone()),
METADATA_COLUMN => Ok(metadata_array.clone()),
name => match metadata_index::indexed_key(field) {
Some(indexed) => Ok(metadata_index::column_from_json(
json.iter().map(Option::as_ref),
&indexed.key,
)),
None => bail!("the table has a column {name} that PutVectors cannot fill"),
},
})
.collect::<Result<Vec<_>>>()?;
RecordBatch::try_new(schema, columns).map_err(Into::into)
}
/// Upserts `vectors` by key into `dataset`, the version the request opened,
/// then fills the indexed-key columns a concurrent column add left empty.
pub(crate) async fn write_vectors(
dataset: Dataset,
vectors: &[pb::Vector],
) -> Result<Dataset, Status> {
let written = Arc::new(Into::<arrow_schema::Schema>::into(dataset.schema()));
let dataset = merge_vectors(dataset, vectors).await?;
let keys: Vec<String> = vectors.iter().map(|v| v.key.clone()).collect();
heal_columns(dataset, &written, &keys)
.await
.map_err(internal)
}
/// The upsert itself, in the schema of the version `dataset` is at.
pub(crate) async fn merge_vectors(
dataset: Dataset,
vectors: &[pb::Vector],
) -> Result<Dataset, Status> {
let dimension = VectorTable::resolve(&dataset, None, Some(KEY_COLUMN))
.map_err(invalid)?
.dimension;
let batch = batch_of(
vectors,
dimension,
Arc::new(Into::<arrow_schema::Schema>::into(dataset.schema())),
)
.map_err(Into::into)
.map_err(invalid)?;
// Upsert by key in one merge commit: no window where the old rows are
// already deleted, and a losing writer fails the request instead of
// leaving duplicate keys.
let mut builder = MergeInsertBuilder::try_new(Arc::new(dataset), vec![KEY_COLUMN.to_string()])
.map_err(internal)?;
builder.when_matched(WhenMatched::UpdateAll);
builder.when_not_matched(WhenNotMatched::InsertAll);
let (dataset, _stats) = builder
.try_build()
.map_err(internal)?
.execute_batches(vec![batch])
.await
.map_err(internal)?;
Ok(Arc::unwrap_or_clone(dataset))
}
/// Attempts [`heal_columns`] makes against writes committing meanwhile.
const HEAL_ATTEMPTS: usize = 10;
/// Lance rebases a write onto a column add committed after the write opened
/// its version: the rows it inserted hold NULL in the new column and the
/// rows it updated keep the value derived from their old metadata. Derive
/// such columns again for the rows of `keys` from their stored metadata.
pub(crate) async fn heal_columns(
mut dataset: Dataset,
written: &Schema,
keys: &[String],
) -> Result<Dataset> {
for _ in 0..HEAL_ATTEMPTS {
let schema: Schema = dataset.schema().into();
let missing: Vec<metadata_index::IndexedKey> = metadata_index::indexed_keys(&schema)
.into_iter()
.filter(|k| written.field_with_name(&k.column).is_err())
.collect();
if missing.is_empty() || keys.is_empty() {
return Ok(dataset);
}
let mut scanner = dataset.scan();
scanner
.filter(&key_list(keys))?
.project(&[KEY_COLUMN, METADATA_COLUMN])?;
let rows: Vec<RecordBatch> = scanner.try_into_stream().await?.try_collect().await?;
let mut fields = vec![schema.field_with_name(KEY_COLUMN)?.clone()];
for k in &missing {
fields.push(schema.field_with_name(&k.column)?.clone());
}
let patch_schema = Arc::new(Schema::new(fields));
let mut patches = Vec::new();
for batch in rows.iter().filter(|b| b.num_rows() > 0) {
let (Some(keys), Some(metadata)) = (
batch.column_by_name(KEY_COLUMN),
batch.column_by_name(METADATA_COLUMN),
) else {
bail!("the scan did not return keys and metadata");
};
let mut columns = vec![keys.clone()];
for k in &missing {
columns.push(metadata_index::column_from_metadata(
metadata.as_ref(),
&k.key,
)?);
}
patches.push(RecordBatch::try_new(patch_schema.clone(), columns)?);
}
if patches.is_empty() {
return Ok(dataset);
}
// Lance's own retry would join these values onto rows a later write
// replaced; derive them again from the version that write left.
let mut builder =
MergeInsertBuilder::try_new(Arc::new(dataset.clone()), vec![KEY_COLUMN.to_string()])?;
builder
.when_matched(WhenMatched::UpdateAll)
.when_not_matched(WhenNotMatched::DoNothing)
.conflict_retries(0);
match builder.try_build()?.execute_batches(patches).await {
Ok((healed, _stats)) => {
tracing::info!(
rows = rows.iter().map(|b| b.num_rows()).sum::<usize>(),
"derived indexed metadata key columns a concurrent column add left stale"
);
return Ok(Arc::unwrap_or_clone(healed));
}
Err(lance::Error::TooMuchWriteContention { .. }) => dataset.checkout_latest().await?,
Err(e) => return Err(e.into()),
}
}
bail!("writes kept replacing the rows while their indexed key columns were derived")
}
/// The indexed metadata keys `dataset` records, requested ones first.
async fn metadata_keys(dataset: &Dataset) -> Result<pb::MetadataKeysResponse> {
let rows: u64 = dataset
.fragments()
.iter()
.map(|f| f.num_rows().unwrap_or(0) as u64)
.sum();
let mut keys: Vec<pb::MetadataKey> = metadata_index::requests(dataset)
.into_iter()
.map(|key| pb::MetadataKey {
key,
state: "requested".to_string(),
rows,
..Default::default()
})
.collect();
for key in metadata_index::keys_of(dataset) {
let segments = metadata_index::segments(dataset, &key.column).await?;
keys.push(pb::MetadataKey {
state: key.state.map_or("unknown", |s| s.as_str()).to_string(),
indexed_rows: segments.iter().map(|s| s.rows).sum(),
index_bytes: segments.iter().map(|s| s.bytes).sum(),
segments: segments.len() as u32,
rows,
key: key.key,
column: key.column,
});
}
let vector_index = match index::index_names(dataset, VECTOR_COLUMN).await?.first() {
Some(name) => {
weed_lance_worker::vector_index::index_settings(dataset, name)
.await?
.index_type
}
None => String::new(),
};
Ok(pb::MetadataKeysResponse {
keys,
version: dataset.version().version,
rows,
vector_index,
})
}
fn struct_to_json(s: &Struct) -> serde_json::Value {
@@ -230,7 +408,7 @@ fn json_to_value(v: serde_json::Value) -> Value {
Value { kind: Some(kind) }
}
fn json_to_struct(map: serde_json::Map<String, serde_json::Value>) -> Struct {
pub(crate) fn json_to_struct(map: serde_json::Map<String, serde_json::Value>) -> Struct {
Struct {
fields: map
.into_iter()
@@ -528,6 +706,8 @@ pub struct VectorServer {
lance_cache_bytes: (usize, usize),
leases: SharedLeases,
nprobes: Option<usize>,
exact_match_limit: Option<usize>,
filter_stats: crate::filter_stats::FilterStats,
/// URI schemes a table's location may have.
schemes: &'static [&'static str],
}
@@ -551,6 +731,8 @@ impl VectorServer {
handles: moka::sync::Cache::new(MAX_HANDLES),
leases: SharedLeases::default(),
nprobes: None,
exact_match_limit: None,
filter_stats: Default::default(),
schemes: &["s3"],
}
}
@@ -568,6 +750,13 @@ impl VectorServer {
self
}
/// Matches of a filter's indexed metadata keys scored exactly instead of
/// searched through the vector index, at most; None keeps the default.
pub fn with_exact_match_limit(mut self, limit: Option<usize>) -> Self {
self.exact_match_limit = limit;
self
}
/// Keep tables on the filers and volume servers, not through the gateway.
/// Unset only for tests on an in-process store.
pub fn with_storage(mut self, storage: DirectReads) -> Self {
@@ -740,7 +929,11 @@ impl VectorService for VectorServer {
)
.await;
let dataset = match dataset {
Ok(dataset) => dataset,
Ok(dataset) => {
self.filter_stats
.forget(&table.id.join("/"), table_id(&dataset));
dataset
}
Err(error) => match self.open(&table).await {
Ok(existing)
if matches_index_schema(
@@ -791,6 +984,7 @@ impl VectorService for VectorServer {
store.remove_dir_all(path).await.map_err(internal)?;
}
}
self.filter_stats.forget(&table.id.join("/"), "");
Ok(Response::new(pb::DropIndexResponse {}))
}
@@ -802,35 +996,37 @@ impl VectorService for VectorServer {
let table = req
.table
.ok_or_else(|| Status::invalid_argument("missing table"))?;
let dataset = self.open(&table).await?;
if req.vectors.is_empty() {
return Ok(Response::new(pb::PutVectorsResponse {
version: dataset.version().version,
}));
if table.version != 0 {
return Err(Status::invalid_argument("a write takes no table version"));
}
let dimension = VectorTable::resolve(&dataset, None, Some(KEY_COLUMN))
.map_err(invalid)?
.dimension;
let batch = batch_of(
&req.vectors,
dimension,
Arc::new(Into::<arrow_schema::Schema>::into(dataset.schema())),
)
.map_err(invalid)?;
// Upsert by key in one merge commit: no window where the old rows are
// already deleted, and a losing writer fails the request instead of
// leaving duplicate keys.
let mut builder =
MergeInsertBuilder::try_new(Arc::new(dataset), vec![KEY_COLUMN.to_string()])
.map_err(internal)?;
builder.when_matched(WhenMatched::UpdateAll);
builder.when_not_matched(WhenNotMatched::InsertAll);
let (dataset, _stats) = builder
.try_build()
.map_err(internal)?
.execute_batches(vec![batch])
.await
.map_err(internal)?;
// A write commits within its deadline of opening the table or not
// at all: a key's column add waits longer than that before the key
// is ready, so a write that opened before the add has committed,
// and been derived again, before any query uses the key.
let merged = tokio::time::timeout(metadata_index::WRITE_DEADLINE, async {
let dataset = self.open(&table).await?;
if req.vectors.is_empty() {
return Ok::<_, Status>((dataset, None));
}
let written = Arc::new(Into::<arrow_schema::Schema>::into(dataset.schema()));
Ok((merge_vectors(dataset, &req.vectors).await?, Some(written)))
})
.await
.map_err(|_| {
Status::deadline_exceeded(format!(
"PutVectors took longer than {:?} to commit",
metadata_index::WRITE_DEADLINE
))
})??;
let dataset = match merged {
(dataset, Some(written)) => {
let keys: Vec<String> = req.vectors.iter().map(|v| v.key.clone()).collect();
heal_columns(dataset, &written, &keys)
.await
.map_err(internal)?
}
(dataset, None) => dataset,
};
Ok(Response::new(pb::PutVectorsResponse {
version: dataset.version().version,
}))
@@ -908,14 +1104,9 @@ impl VectorService for VectorServer {
0 => DEFAULT_PAGE,
n => n.min(MAX_PAGE),
};
// A purge keeps only the latest version, so lease the one this page reads.
let _lease = self.leases.try_pin(&dataset).await.map_err(internal)?;
if segments > 1 {
// Data and metadata come from a second read of this version, so
// lease it as get_vectors does.
let _lease = if columns.len() > 1 {
self.leases.try_pin(&dataset).await.map_err(internal)?
} else {
None
};
let (vectors, next) = list_segment(
&dataset,
&columns,
@@ -953,6 +1144,7 @@ impl VectorService for VectorServer {
&self,
request: Request<pb::QueryVectorsRequest>,
) -> Result<Response<pb::QueryVectorsResponse>, Status> {
let started = std::time::Instant::now();
let req = request.into_inner();
let table = req
.table
@@ -962,6 +1154,11 @@ impl VectorService for VectorServer {
// while this query runs against it.
let _lease = self.leases.try_pin(&dataset).await.map_err(internal)?;
let resolved = VectorTable::resolve(&dataset, None, Some(KEY_COLUMN)).map_err(invalid)?;
let filter_keys = if req.filter.is_empty() {
Default::default()
} else {
indexed_filter::filter_keys(&dataset, &req.filter).unwrap_or_default()
};
if req.query_vector.len() != resolved.dimension {
return Err(Status::invalid_argument(format!(
"query vector carries {} floats, the index dimension is {}",
@@ -986,6 +1183,7 @@ impl VectorService for VectorServer {
nprobes: self.nprobes,
index_segments: None,
fragments: None,
exact_match_limit: self.exact_match_limit,
};
let hits = query::ann(&dataset, &resolved, &query)
.await
@@ -1061,8 +1259,59 @@ impl VectorService for VectorServer {
data: data_map.remove(&hit.key).unwrap_or_default(),
});
}
let index = table.id.join("/");
self.filter_stats
.record(&index, table_id(&dataset), &filter_keys, started.elapsed());
Ok(Response::new(pb::QueryVectorsResponse { vectors }))
}
async fn metadata_keys(
&self,
request: Request<pb::MetadataKeysRequest>,
) -> Result<Response<pb::MetadataKeysResponse>, Status> {
let table = request
.into_inner()
.table
.ok_or_else(|| Status::invalid_argument("missing table"))?;
let dataset = self.open_to_read(&table).await?;
let _lease = self.leases.try_pin(&dataset).await.map_err(internal)?;
Ok(Response::new(
metadata_keys(&dataset).await.map_err(internal)?,
))
}
async fn update_metadata_key(
&self,
request: Request<pb::UpdateMetadataKeyRequest>,
) -> Result<Response<pb::MetadataKeysResponse>, Status> {
let req = request.into_inner();
let table = req
.table
.ok_or_else(|| Status::invalid_argument("missing table"))?;
let mut dataset = self.open(&table).await?;
let updated = if req.remove {
metadata_index::request_removal(&mut dataset, &req.key).await
} else {
metadata_index::request(&mut dataset, &req.key).await
};
updated.map_err(|e| Status::failed_precondition(format!("{e:#}")))?;
Ok(Response::new(
metadata_keys(&dataset).await.map_err(internal)?,
))
}
async fn filter_stats(
&self,
request: Request<pb::FilterStatsRequest>,
) -> Result<Response<pb::FilterStatsResponse>, Status> {
let table = request
.into_inner()
.table
.ok_or_else(|| Status::invalid_argument("missing table"))?;
Ok(Response::new(
self.filter_stats.snapshot(&table.id.join("/")),
))
}
}
#[cfg(test)]
+1
View File
@@ -147,6 +147,7 @@ service VolumeServer {
message BatchDeleteRequest {
repeated string file_ids = 1;
bool skip_cookie_check = 2;
bool fsync = 3;
}
message BatchDeleteResponse {
+61
View File
@@ -306,6 +306,8 @@ pub struct VolumeServerConfig {
pub ui_enabled: bool,
pub jwt_signing_key: Vec<u8>,
pub jwt_signing_expires_seconds: i64,
pub jwt_previous_signing_key: Vec<u8>,
pub jwt_previous_read_signing_key: Vec<u8>,
pub jwt_read_signing_key: Vec<u8>,
pub jwt_read_signing_expires_seconds: i64,
pub https_cert_file: String,
@@ -999,6 +1001,8 @@ fn resolve_config_with_env(cli: Cli, env: EnvLookup<'_>) -> VolumeServerConfig {
ui_enabled: sec.jwt_signing_key.is_empty() || sec.access_ui,
jwt_signing_key: sec.jwt_signing_key,
jwt_signing_expires_seconds: sec.jwt_signing_expires,
jwt_previous_signing_key: sec.jwt_previous_signing_key,
jwt_previous_read_signing_key: sec.jwt_previous_read_signing_key,
jwt_read_signing_key: sec.jwt_read_signing_key,
jwt_read_signing_expires_seconds: sec.jwt_read_signing_expires,
https_cert_file: sec.https_cert_file,
@@ -1030,6 +1034,8 @@ fn default_volume_dir() -> String {
pub struct SecurityConfig {
pub jwt_signing_key: Vec<u8>,
pub jwt_signing_expires: i64,
pub jwt_previous_signing_key: Vec<u8>,
pub jwt_previous_read_signing_key: Vec<u8>,
pub jwt_read_signing_key: Vec<u8>,
pub jwt_read_signing_expires: i64,
pub https_cert_file: String,
@@ -1048,6 +1054,8 @@ pub struct SecurityConfig {
pub grpc_volume_allowed_common_names: Vec<String>,
pub tls_policy: TlsPolicy,
pub access_ui: bool,
/// The file was read; otherwise only defaults and WEED_ overrides are set.
pub file_read: bool,
/// IPs from [guard] white_list in security.toml
pub guard_white_list: Vec<String>,
}
@@ -1180,6 +1188,7 @@ fn parse_security_config_with_env(path: &str, env: EnvLookup<'_>) -> SecurityCon
match section {
Section::JwtSigningRead => match key {
"key" => cfg.jwt_read_signing_key = value.as_bytes().to_vec(),
"previous_key" => cfg.jwt_previous_read_signing_key = value.as_bytes().to_vec(),
"expires_after_seconds" => {
cfg.jwt_read_signing_expires = value.parse().unwrap_or(60)
}
@@ -1187,6 +1196,7 @@ fn parse_security_config_with_env(path: &str, env: EnvLookup<'_>) -> SecurityCon
},
Section::JwtSigning => match key {
"key" => cfg.jwt_signing_key = value.as_bytes().to_vec(),
"previous_key" => cfg.jwt_previous_signing_key = value.as_bytes().to_vec(),
"expires_after_seconds" => {
cfg.jwt_signing_expires = value.parse().unwrap_or(10)
}
@@ -1262,6 +1272,7 @@ fn parse_security_config_with_env(path: &str, env: EnvLookup<'_>) -> SecurityCon
// prefix WEED_, uppercase, replace . with _).
// e.g. WEED_JWT_SIGNING_KEY overrides [jwt.signing] key
apply_env_overrides(&mut cfg, env);
cfg.file_read = true;
cfg
}
@@ -1300,6 +1311,12 @@ fn apply_env_overrides(cfg: &mut SecurityConfig, env: EnvLookup<'_>) {
if let Some(v) = env_string(env, "WEED_JWT_SIGNING_EXPIRES_AFTER_SECONDS") {
cfg.jwt_signing_expires = v.parse().unwrap_or(cfg.jwt_signing_expires);
}
if let Some(v) = env_string(env, "WEED_JWT_SIGNING_PREVIOUS_KEY") {
cfg.jwt_previous_signing_key = v.into_bytes();
}
if let Some(v) = env_string(env, "WEED_JWT_SIGNING_READ_PREVIOUS_KEY") {
cfg.jwt_previous_read_signing_key = v.into_bytes();
}
if let Some(v) = env_string(env, "WEED_JWT_SIGNING_READ_KEY") {
cfg.jwt_read_signing_key = v.into_bytes();
}
@@ -2177,6 +2194,50 @@ cipher_suites = "TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256"
assert_eq!(cfg.jwt_signing_key, b"env-secret");
}
#[test]
fn test_previous_signing_keys_from_file_and_env() {
let tmp = tempfile::NamedTempFile::new().unwrap();
std::fs::write(
tmp.path(),
r#"
[jwt.signing]
key = "write-new"
previous_key = "write-old"
[jwt.signing.read]
key = "read-new"
previous_key = "read-old"
"#,
)
.unwrap();
let path = tmp.path().to_str().unwrap();
let cfg = parse_security_config_with_env(path, &fake_env(&[]));
assert_eq!(cfg.jwt_previous_signing_key, b"write-old");
assert_eq!(cfg.jwt_previous_read_signing_key, b"read-old");
let env = fake_env(&[
("WEED_JWT_SIGNING_PREVIOUS_KEY", "env-write-old"),
("WEED_JWT_SIGNING_READ_PREVIOUS_KEY", "env-read-old"),
]);
let cfg = parse_security_config_with_env(path, &env);
assert_eq!(cfg.jwt_previous_signing_key, b"env-write-old");
assert_eq!(cfg.jwt_previous_read_signing_key, b"env-read-old");
}
#[test]
fn test_security_config_file_read() {
let tmp = tempfile::NamedTempFile::new().unwrap();
std::fs::write(tmp.path(), "[jwt.signing]\nkey = \"secret\"\n").unwrap();
let path = tmp.path().to_str().unwrap().to_string();
assert!(parse_security_config_with_env(&path, &fake_env(&[])).file_read);
drop(tmp);
let env = fake_env(&[("WEED_JWT_SIGNING_KEY", "env-secret")]);
let cfg = parse_security_config_with_env(&path, &env);
assert!(!cfg.file_read);
assert_eq!(cfg.jwt_signing_key, b"env-secret");
}
#[test]
fn test_env_override_takes_precedence_over_file() {
let tmp = tempfile::NamedTempFile::new().unwrap();
+11 -2
View File
@@ -340,7 +340,7 @@ async fn run(
store.set_volume_load_guard(volume_copies_in_flight.clone());
// Build shared state
let guard = Guard::new(
let mut guard = Guard::new(
&config.white_list,
SigningKey(config.jwt_signing_key.clone()),
config.jwt_signing_expires_seconds,
@@ -358,6 +358,8 @@ async fn run(
let security_file = config.security_file.clone();
let cli_white_list = config.white_list.clone();
guard.previous_signing_key = SigningKey(config.jwt_previous_signing_key.clone());
guard.previous_read_signing_key = SigningKey(config.jwt_previous_read_signing_key.clone());
let state = Arc::new(VolumeServerState {
store: RwLock::new(store),
@@ -416,6 +418,7 @@ async fn run(
.to_string_lossy()
.into_owned()
},
ec_lifecycle_locks: std::sync::Mutex::new(std::collections::HashMap::new()),
ec_decodes_in_flight: std::sync::Mutex::new(std::collections::HashSet::new()),
volume_copies_in_flight,
ec_decode_tail: std::sync::Mutex::new(std::collections::HashSet::new()),
@@ -618,9 +621,13 @@ async fn run(
}
// 2. Reload security config (Go's Reload)
{
'security: {
info!("Reloading security config...");
let sec = config::parse_security_config(&state_reload.security_file);
if !sec.file_read {
warn!("SIGHUP: security config cannot be read, keeping the keys in effect");
break 'security;
}
let mut whitelist = state_reload.cli_white_list.clone();
whitelist.extend(sec.guard_white_list.iter().cloned());
let mut guard = state_reload.guard.write().unwrap();
@@ -630,6 +637,8 @@ async fn run(
sec.jwt_signing_expires,
SigningKey(sec.jwt_read_signing_key),
sec.jwt_read_signing_expires,
SigningKey(sec.jwt_previous_signing_key),
SigningKey(sec.jwt_previous_read_signing_key),
);
}
+14
View File
@@ -85,6 +85,19 @@ pub static VACUUMING_COMMIT_COUNTER: LazyLock<IntCounterVec> = LazyLock::new(||
.expect("metric can be created")
});
/// Live needles vacuum dropped because their data could not be read back
/// intact, with label `collection`.
pub static VACUUMING_DROPPED_NEEDLES_COUNTER: LazyLock<IntCounterVec> = LazyLock::new(|| {
IntCounterVec::new(
Opts::new(
"SeaweedFS_volumeServer_vacuuming_dropped_needles",
"Counter of live needles dropped by vacuum because their data could not be read back intact.",
),
&["collection"],
)
.expect("metric can be created")
});
/// Vacuuming duration histogram with label `type` (compact/commit).
pub static VACUUMING_HISTOGRAM: LazyLock<HistogramVec> = LazyLock::new(|| {
HistogramVec::new(
@@ -402,6 +415,7 @@ pub fn register_metrics() {
Box::new(VACUUMING_COMPACT_COUNTER.clone()),
Box::new(VACUUMING_COMMIT_COUNTER.clone()),
Box::new(VACUUMING_HISTOGRAM.clone()),
Box::new(VACUUMING_DROPPED_NEEDLES_COUNTER.clone()),
Box::new(VOLUME_GAUGE.clone()),
Box::new(READ_ONLY_VOLUME_GAUGE.clone()),
Box::new(MAX_VOLUMES.clone()),
+207 -23
View File
@@ -83,7 +83,13 @@ pub fn gen_jwt(
}
/// Decode and validate a JWT token.
pub fn decode_jwt(signing_key: &SigningKey, token: &str) -> Result<FileIdClaims, JwtError> {
/// A token whose signature does not match `signing_key` is checked against
/// each non-empty `previous_keys` entry, like Go's `security.DecodeJwt`.
pub fn decode_jwt(
signing_key: &SigningKey,
token: &str,
previous_keys: &[&SigningKey],
) -> Result<FileIdClaims, JwtError> {
if signing_key.is_empty() {
return Err(JwtError::NoSigningKey);
}
@@ -97,13 +103,64 @@ pub fn decode_jwt(signing_key: &SigningKey, token: &str) -> Result<FileIdClaims,
validation.validate_nbf = true;
validation.leeway = 0;
let data = decode::<FileIdClaims>(
let mut result = decode::<FileIdClaims>(
token,
&DecodingKey::from_secret(&signing_key.0),
&validation,
)?;
);
for previous in previous_keys {
let bad_signature = matches!(&result, Err(e)
if matches!(e.kind(), jsonwebtoken::errors::ErrorKind::InvalidSignature));
if !bad_signature {
break;
}
if previous.is_empty() || previous.0 == signing_key.0 {
continue;
}
result = decode::<FileIdClaims>(token, &DecodingKey::from_secret(&previous.0), &validation);
}
Ok(data.claims)
Ok(result?.claims)
}
/// Reports a signed, fid-bound write token whose only failure is expiry. It
/// never authorizes a request. Mirrors Go's `security.IsExpiredVolumeJwt`.
pub fn is_expired_volume_jwt(
signing_key: &SigningKey,
token: &str,
expected_fid: &str,
previous_keys: &[&SigningKey],
) -> bool {
match decode_jwt(signing_key, token, previous_keys) {
Err(JwtError::Jwt(e))
if matches!(e.kind(), jsonwebtoken::errors::ErrorKind::ExpiredSignature) => {}
_ => return false,
}
#[derive(Deserialize)]
struct ExpiredClaims {
fid: Option<String>,
iat: Option<u64>,
}
let mut validation = Validation::new(Algorithm::HS256);
validation.required_spec_claims.clear();
validation.validate_exp = false;
validation.validate_nbf = true;
validation.leeway = 0;
let Some(data) = std::iter::once(signing_key)
.chain(previous_keys.iter().copied())
.filter(|key| !key.is_empty())
.find_map(|key| {
decode::<ExpiredClaims>(token, &DecodingKey::from_secret(&key.0), &validation).ok()
})
else {
return false;
};
let now = SystemTime::now()
.duration_since(UNIX_EPOCH)
.unwrap_or_default()
.as_secs();
data.claims.fid.as_deref() == Some(expected_fid) && data.claims.iat.is_none_or(|iat| iat <= now)
}
// ============================================================================
@@ -118,6 +175,9 @@ pub struct Guard {
pub expires_after_sec: i64,
pub read_signing_key: SigningKey,
pub read_expires_after_sec: i64,
/// Accepted when verifying, never used to sign.
pub previous_signing_key: SigningKey,
pub previous_read_signing_key: SigningKey,
/// Combined flag: true when whitelist is non-empty OR signing key is present.
/// Matches Go's `isWriteActive = !isEmptyWhiteList || len(SigningKey) != 0`.
is_write_active: bool,
@@ -138,6 +198,8 @@ impl Guard {
expires_after_sec,
read_signing_key,
read_expires_after_sec,
previous_signing_key: SigningKey(vec![]),
previous_read_signing_key: SigningKey(vec![]),
is_write_active: false,
};
guard.update_whitelist(whitelist);
@@ -182,11 +244,15 @@ impl Guard {
expires_after_sec: i64,
read_signing_key: SigningKey,
read_expires_after_sec: i64,
previous_signing_key: SigningKey,
previous_read_signing_key: SigningKey,
) {
self.signing_key = signing_key;
self.expires_after_sec = expires_after_sec;
self.read_signing_key = read_signing_key;
self.read_expires_after_sec = read_expires_after_sec;
self.previous_signing_key = previous_signing_key;
self.previous_read_signing_key = previous_read_signing_key;
let is_empty_whitelist = self.whitelist_ips.is_empty() && self.whitelist_cidrs.is_empty();
self.is_write_active = !is_empty_whitelist || !self.signing_key.is_empty();
@@ -229,6 +295,16 @@ impl Guard {
false
}
/// The write (or read) signing key and the previous key still accepted for
/// it. Mirrors Go's `Guard.VerificationKeys`.
pub fn verification_keys(&self, is_write: bool) -> (&SigningKey, &SigningKey) {
if is_write {
(&self.signing_key, &self.previous_signing_key)
} else {
(&self.read_signing_key, &self.previous_read_signing_key)
}
}
/// Check if a read signing key is configured.
pub fn has_read_signing_key(&self) -> bool {
!self.read_signing_key.is_empty()
@@ -238,18 +314,14 @@ impl Guard {
/// `is_write` determines which signing key to use.
/// Returns Ok(()) if valid, or if security is disabled.
pub fn check_jwt(&self, token: Option<&str>, is_write: bool) -> Result<(), JwtError> {
let key = if is_write {
&self.signing_key
} else {
&self.read_signing_key
};
let (key, previous) = self.verification_keys(is_write);
if key.is_empty() {
return Ok(()); // Security disabled for this operation type
}
let token = token.ok_or(JwtError::MissingToken)?;
decode_jwt(key, token)?;
decode_jwt(key, token, &[previous])?;
Ok(())
}
@@ -260,18 +332,14 @@ impl Guard {
expected_fid: &str,
is_write: bool,
) -> Result<(), JwtError> {
let key = if is_write {
&self.signing_key
} else {
&self.read_signing_key
};
let (key, previous) = self.verification_keys(is_write);
if key.is_empty() {
return Ok(());
}
let token = token.ok_or(JwtError::MissingToken)?;
let claims = decode_jwt(key, token)?;
let claims = decode_jwt(key, token, &[previous])?;
match claims.fid {
None => {
@@ -385,7 +453,7 @@ mod tests {
fn test_jwt_round_trip() {
let key = SigningKey::from_string("test-secret-key");
let token = gen_jwt(&key, 3600, "3,01637037d6").unwrap();
let claims = decode_jwt(&key, &token).unwrap();
let claims = decode_jwt(&key, &token, &[]).unwrap();
assert_eq!(claims.fid, Some("3,01637037d6".to_string()));
}
@@ -398,7 +466,7 @@ mod tests {
#[test]
fn test_jwt_invalid_token() {
let key = SigningKey::from_string("secret");
let result = decode_jwt(&key, "invalid.token.here");
let result = decode_jwt(&key, "invalid.token.here", &[]);
assert!(result.is_err());
}
@@ -407,7 +475,60 @@ mod tests {
let key1 = SigningKey::from_string("secret1");
let key2 = SigningKey::from_string("secret2");
let token = gen_jwt(&key1, 3600, "1,abc").unwrap();
assert!(decode_jwt(&key2, &token).is_err());
assert!(decode_jwt(&key2, &token, &[]).is_err());
}
#[test]
fn test_is_expired_volume_jwt_classifies_expiry_only() {
let key = SigningKey::from_string("write-key");
let fid = "3,01637037d6";
let now = SystemTime::now()
.duration_since(UNIX_EPOCH)
.unwrap()
.as_secs();
let sign = |claims: serde_json::Value, key: &SigningKey| {
encode(
&Header::new(Algorithm::HS256),
&claims,
&EncodingKey::from_secret(&key.0),
)
.unwrap()
};
let expired = sign(serde_json::json!({"fid": fid, "exp": now - 10}), &key);
assert!(is_expired_volume_jwt(&key, &expired, fid, &[]));
assert!(!is_expired_volume_jwt(&key, &expired, "3,01637037d7", &[]));
assert!(!is_expired_volume_jwt(
&SigningKey::from_string("other-key"),
&expired,
fid,
&[]
));
assert!(!is_expired_volume_jwt(
&SigningKey(vec![]),
&expired,
fid,
&[]
));
let immature = sign(
serde_json::json!({"fid": fid, "exp": now - 10, "nbf": now + 60}),
&key,
);
assert!(!is_expired_volume_jwt(&key, &immature, fid, &[]));
let future_issued = sign(
serde_json::json!({"fid": fid, "exp": now - 10, "iat": now + 60}),
&key,
);
assert!(!is_expired_volume_jwt(&key, &future_issued, fid, &[]));
let live = sign(serde_json::json!({"fid": fid, "exp": now + 60}), &key);
assert!(!is_expired_volume_jwt(&key, &live, fid, &[]));
let unsigned = format!(
"eyJhbGciOiJub25lIiwidHlwIjoiSldUIn0.{}.",
base64::Engine::encode(
&base64::engine::general_purpose::URL_SAFE_NO_PAD,
serde_json::json!({"fid": fid, "exp": now - 10}).to_string()
)
);
assert!(!is_expired_volume_jwt(&key, &unsigned, fid, &[]));
}
#[test]
@@ -507,25 +628,88 @@ mod tests {
11,
SigningKey::from_string("new-read"),
61,
SigningKey(vec![]),
SigningKey(vec![]),
);
let token = gen_jwt(&guard.signing_key, 60, "3,01637037d6").unwrap();
// Validates with the rotated key, not the old one.
assert!(decode_jwt(&guard.signing_key, &token).is_ok());
assert!(decode_jwt(&SigningKey::from_string("old-write"), &token).is_err());
assert!(decode_jwt(&guard.signing_key, &token, &[]).is_ok());
assert!(decode_jwt(&SigningKey::from_string("old-write"), &token, &[]).is_err());
assert_eq!(guard.expires_after_sec, 11);
assert_eq!(guard.read_expires_after_sec, 61);
}
#[test]
fn test_previous_signing_key_is_accepted() {
let fid = "3,01637037d6";
let accepts = |guard: &Guard, key: &str, is_write: bool| {
let token = gen_jwt(&SigningKey::from_string(key), 60, fid).unwrap();
guard
.check_jwt_for_file(Some(&token), fid, is_write)
.is_ok()
};
let mut guard = Guard::new(
&[],
SigningKey::from_string("write-new"),
10,
SigningKey::from_string("read-new"),
60,
);
guard.previous_signing_key = SigningKey::from_string("write-old");
guard.previous_read_signing_key = SigningKey::from_string("read-old");
assert!(accepts(&guard, "write-new", true));
assert!(accepts(&guard, "write-old", true));
assert!(accepts(&guard, "read-old", false));
assert!(!accepts(&guard, "other", true));
assert!(!accepts(&guard, "read-old", true));
let expired = encode(
&Header::new(Algorithm::HS256),
&serde_json::json!({"fid": fid, "exp": 1}),
&EncodingKey::from_secret(b"write-old"),
)
.unwrap();
let (key, previous) = guard.verification_keys(true);
assert!(is_expired_volume_jwt(key, &expired, fid, &[previous]));
assert!(!is_expired_volume_jwt(key, &expired, fid, &[]));
guard.update_signing_keys(
SigningKey::from_string("write-new"),
10,
SigningKey::from_string("read-new"),
60,
SigningKey(vec![]),
SigningKey(vec![]),
);
assert!(!accepts(&guard, "write-old", true));
assert!(!accepts(&guard, "read-old", false));
}
#[test]
fn test_update_signing_keys_toggles_write_active() {
let mut guard = Guard::new(&[], SigningKey(vec![]), 0, SigningKey(vec![]), 0);
assert!(guard.check_jwt(None, true).is_ok());
guard.update_signing_keys(SigningKey::from_string("write"), 10, SigningKey(vec![]), 0);
guard.update_signing_keys(
SigningKey::from_string("write"),
10,
SigningKey(vec![]),
0,
SigningKey(vec![]),
SigningKey(vec![]),
);
assert!(guard.is_write_active);
guard.update_signing_keys(SigningKey(vec![]), 0, SigningKey(vec![]), 0);
guard.update_signing_keys(
SigningKey(vec![]),
0,
SigningKey(vec![]),
0,
SigningKey(vec![]),
SigningKey(vec![]),
);
assert!(!guard.is_write_active);
}
+15 -2
View File
@@ -110,6 +110,17 @@ pub fn load_outgoing_grpc_tls(
}))
}
#[cfg(test)]
impl OutgoingGrpcTlsConfig {
pub(crate) fn self_signed_for_test() -> Self {
Self {
cert_pem: tests::TEST_CERT_PEM.to_string(),
key_pem: tests::TEST_KEY_PEM.to_string(),
ca_pem: tests::TEST_CERT_PEM.to_string(),
}
}
}
pub fn grpc_endpoint_uri(grpc_host_port: &str, tls: Option<&OutgoingGrpcTlsConfig>) -> String {
let scheme = if tls.is_some() { "https" } else { "http" };
format!("{}://{}", scheme, grpc_host_port)
@@ -360,8 +371,8 @@ mod tests {
use std::sync::{Arc, Mutex};
use std::time::Duration;
const TEST_CERT_PEM: &str = "-----BEGIN CERTIFICATE-----\nMIIBPDCB76ADAgECAhRuRPQgeAu43BT/M7EfAWSdapVdYDAFBgMrZXAwFDESMBAG\nA1UEAwwJbG9jYWxob3N0MB4XDTI2MDcwNTE2MTUyOVoXDTM2MDcwMjE2MTUyOVow\nFDESMBAGA1UEAwwJbG9jYWxob3N0MCowBQYDK2VwAyEAr/3bNIFI+8V32oCiY6y+\nXRFmZpdNQ2g//VtRkT+nQg+jUzBRMB0GA1UdDgQWBBTsy9tLf1zPiXCQfgci6zNi\ndEzRSjAfBgNVHSMEGDAWgBTsy9tLf1zPiXCQfgci6zNidEzRSjAPBgNVHRMBAf8E\nBTADAQH/MAUGAytlcANBAIvsdw0IbvOBBkb9cd7BfMJfIP9pQQrAL03pCRWJFnFh\nSysaLVgFXI4T078IiaM874oO+iB+5vNbWEpc7CkGow4=\n-----END CERTIFICATE-----\n";
const TEST_KEY_PEM: &str = "-----BEGIN PRIVATE KEY-----\nMC4CAQAwBQYDK2VwBCIEIHbyn71Kk+Y7KT3sBctit7uZpErpoH6qDbFj6P8qGaZH\n-----END PRIVATE KEY-----\n";
pub(super) const TEST_CERT_PEM: &str = "-----BEGIN CERTIFICATE-----\nMIIBPDCB76ADAgECAhRuRPQgeAu43BT/M7EfAWSdapVdYDAFBgMrZXAwFDESMBAG\nA1UEAwwJbG9jYWxob3N0MB4XDTI2MDcwNTE2MTUyOVoXDTM2MDcwMjE2MTUyOVow\nFDESMBAGA1UEAwwJbG9jYWxob3N0MCowBQYDK2VwAyEAr/3bNIFI+8V32oCiY6y+\nXRFmZpdNQ2g//VtRkT+nQg+jUzBRMB0GA1UdDgQWBBTsy9tLf1zPiXCQfgci6zNi\ndEzRSjAfBgNVHSMEGDAWgBTsy9tLf1zPiXCQfgci6zNidEzRSjAPBgNVHRMBAf8E\nBTADAQH/MAUGAytlcANBAIvsdw0IbvOBBkb9cd7BfMJfIP9pQQrAL03pCRWJFnFh\nSysaLVgFXI4T078IiaM874oO+iB+5vNbWEpc7CkGow4=\n-----END CERTIFICATE-----\n";
pub(super) const TEST_KEY_PEM: &str = "-----BEGIN PRIVATE KEY-----\nMC4CAQAwBQYDK2VwBCIEIHbyn71Kk+Y7KT3sBctit7uZpErpoH6qDbFj6P8qGaZH\n-----END PRIVATE KEY-----\n";
#[test]
fn test_build_grpc_endpoint_with_tls_resolves_crypto_provider() {
@@ -425,6 +436,8 @@ mod tests {
ui_enabled: false,
jwt_signing_key: vec![],
jwt_signing_expires_seconds: 0,
jwt_previous_signing_key: vec![],
jwt_previous_read_signing_key: vec![],
jwt_read_signing_key: vec![],
jwt_read_signing_expires_seconds: 0,
https_cert_file: String::new(),
File diff suppressed because it is too large. Load diff
File diff suppressed because it is too large. Load diff
+1
View File
@@ -1698,6 +1698,7 @@ mod tests {
security_file: String::new(),
cli_white_list: vec![],
state_file_path: String::new(),
ec_lifecycle_locks: std::sync::Mutex::new(std::collections::HashMap::new()),
ec_decodes_in_flight: std::sync::Mutex::new(std::collections::HashSet::new()),
volume_copies_in_flight: std::sync::Arc::new(std::sync::Mutex::new(std::collections::HashSet::new())),
ec_decode_tail: std::sync::Mutex::new(std::collections::HashSet::new()),
+2
View File
@@ -156,6 +156,8 @@ mod tests {
ui_enabled: false,
jwt_signing_key: vec![],
jwt_signing_expires_seconds: 0,
jwt_previous_signing_key: vec![],
jwt_previous_read_signing_key: vec![],
jwt_read_signing_key: vec![],
jwt_read_signing_expires_seconds: 0,
https_cert_file: String::new(),
+31 -9
View File
@@ -2384,23 +2384,45 @@ pub(crate) async fn recover_missing_ec_indexes(
"cannot look up peers to recover missing .ecx: {}",
e
);
continue;
Vec::new()
}
};
if peers.is_empty() {
tracing::warn!(
volume_id = m.vid.0,
"shards present locally but .ecx missing and no peer holds it; leaving shards unloaded"
);
continue;
// Under the volume's lifecycle lock, so an in-flight VolumeEcShardsCopy
// of it, which may unlink the shards it renamed in, never has them
// mounted from under it.
let _lifecycle = state.ec_lifecycle_lock(m.vid).lock_owned().await;
let current = state
.store
.read()
.unwrap()
.collect_ec_volumes_missing_index_only(Some(m.vid))
.into_iter()
.find(|x| x.vid == m.vid && x.collection == m.collection);
// A copy holding the lock may have installed the index meanwhile: skip
// the fetch, but its shards still need the mount.
if let Some(current) = &current {
if peers.is_empty() {
tracing::warn!(
volume_id = m.vid.0,
"shards present locally but .ecx missing and no peer holds it; leaving shards unloaded"
);
continue;
}
if !fetch_ec_index_from_peers(state, current, &peers).await {
continue;
}
}
if fetch_ec_index_from_peers(state, m, &peers).await {
state
.store
.write()
.unwrap()
.mount_recovered_ec_shards_only(Some(m.vid));
if current.is_some() {
recovered += 1;
}
}
if recovered > 0 {
state.store.write().unwrap().mount_recovered_ec_shards();
tracing::info!(
"recovered missing EC index for {} volume(s) from peers and mounted their shards",
recovered
@@ -125,6 +125,11 @@ pub struct VolumeServerState {
pub cli_white_list: Vec<String>,
/// Path to state.pb file for persisting VolumeServerState across restarts.
pub state_file_path: String,
/// Per-vid EC lifecycle lock, serializing EC ops that install or remove a
/// vid's files. A tokio Mutex so the guard survives the copy's `.await` IO.
pub ec_lifecycle_locks: std::sync::Mutex<
std::collections::HashMap<crate::storage::types::VolumeId, Arc<tokio::sync::Mutex<()>>>,
>,
/// Volumes with an EC decode in flight. A dropped request leaves the
/// blocking job running; this keeps a retry from racing it on the
/// same volume files.
@@ -151,6 +156,20 @@ pub struct VolumeServerState {
}
impl VolumeServerState {
/// Return the per-vid EC lifecycle lock, creating it on first use. Entries
/// are never reaped.
pub fn ec_lifecycle_lock(
&self,
vid: crate::storage::types::VolumeId,
) -> Arc<tokio::sync::Mutex<()>> {
self.ec_lifecycle_locks
.lock()
.unwrap()
.entry(vid)
.or_insert_with(|| Arc::new(tokio::sync::Mutex::new(())))
.clone()
}
/// The master lookups should go to: the leader the heartbeat confirmed,
/// or empty while none is known, in its announced `host:port[.grpcPort]`
/// form. Mirrors Go's `vs.getCurrentMaster()`.
+1
View File
@@ -467,6 +467,7 @@ mod tests {
security_file: String::new(),
cli_white_list: vec![],
state_file_path: String::new(),
ec_lifecycle_locks: std::sync::Mutex::new(std::collections::HashMap::new()),
ec_decodes_in_flight: std::sync::Mutex::new(std::collections::HashSet::new()),
volume_copies_in_flight: std::sync::Arc::new(std::sync::Mutex::new(std::collections::HashSet::new())),
ec_decode_tail: std::sync::Mutex::new(std::collections::HashSet::new()),
@@ -829,6 +829,24 @@ impl DiskLocation {
self.volumes.len()
}
/// Number of local volumes of `collection` that still take writes: not
/// read-only and, when `volume_size_limit` is known (non zero), with a .dat
/// below it. The .dat size is what the master compares with the limit to
/// stop writes. Mirrors Go's WritableCollectionVolumesLen.
pub fn writable_collection_volumes_len(
&self,
collection: &str,
volume_size_limit: u64,
) -> usize {
self.volumes
.values()
.filter(|v| v.collection == collection && !v.has_remote_file() && !v.is_read_only())
.filter(|v| {
volume_size_limit == 0 || v.dat_file_size().unwrap_or(0) < volume_size_limit
})
.count()
}
/// Get all volume IDs, sorted.
pub fn volume_ids(&self) -> Vec<VolumeId> {
let mut ids: Vec<VolumeId> = self.volumes.keys().copied().collect();
+113 -14
View File
@@ -292,12 +292,20 @@ impl Store {
// ---- Volume lifecycle ----
/// Find the location with fewest volumes (load-balance) of the given disk type.
/// Matches Go's FindFreeLocation: accounts for EC shards when computing free slots.
fn find_free_location(&self, disk_type: &DiskType, preallocate: u64) -> Option<usize> {
/// Find the location with the fewest writable volumes of `collection`, so
/// they do not share one disk's write bandwidth, then the most free slots
/// (load-balance) of the given disk type. Matches Go's FindFreeLocation:
/// accounts for EC shards when computing free slots.
fn find_free_location(
&self,
disk_type: &DiskType,
collection: &str,
preallocate: u64,
) -> Option<usize> {
use crate::storage::erasure_coding::ec_shard::DATA_SHARDS_COUNT;
let mut best: Option<(usize, i64)> = None; // (index, effective_free)
let volume_size_limit = self.volume_size_limit.load(Ordering::Relaxed);
let mut best: Option<(usize, usize, i64)> = None; // (index, writable, effective_free)
for (i, loc) in self.locations.iter().enumerate() {
if &loc.disk_type != disk_type {
continue;
@@ -327,11 +335,14 @@ impl Store {
) {
continue;
}
if best.is_none() || effective_free > best.unwrap().1 {
best = Some((i, effective_free));
let writable = loc.writable_collection_volumes_len(collection, volume_size_limit);
if best.is_none_or(|(_, min_writable, max_free)| {
writable < min_writable || (writable == min_writable && effective_free > max_free)
}) {
best = Some((i, writable, effective_free));
}
}
best.map(|(i, _)| i)
best.map(|(i, _, _)| i)
}
/// Find a free location matching a predicate.
@@ -601,7 +612,7 @@ impl Store {
return Err(VolumeError::AlreadyExists);
}
let loc_idx = self
.find_free_location(&disk_type, spec.preallocate)
.find_free_location(&disk_type, spec.collection, spec.preallocate)
.ok_or_else(|| {
VolumeError::Io(io::Error::other(format!(
"no free location for disk type {:?}",
@@ -1042,13 +1053,18 @@ impl Store {
}
/// Delete a needle from a volume.
pub fn delete_volume_needle(&self, vid: VolumeId, n: &mut Needle) -> Result<Size, VolumeError> {
pub fn delete_volume_needle(
&self,
vid: VolumeId,
n: &mut Needle,
fsync: bool,
) -> Result<Size, VolumeError> {
// Match Go's DeleteVolumeNeedle: check noWriteOrDelete before proceeding.
let (_, vol) = self.find_volume(vid).ok_or(VolumeError::NotFound)?;
if vol.is_no_write_or_delete() {
return Err(VolumeError::ReadOnly(vid));
}
vol.delete_needle(n)
vol.delete_needle(n, fsync)
}
// ---- Collection operations ----
@@ -2923,7 +2939,9 @@ mod tests {
cookie: Cookie(0xaa),
..Needle::default()
};
let deleted = store.delete_volume_needle(VolumeId(1), &mut del_n).unwrap();
let deleted = store
.delete_volume_needle(VolumeId(1), &mut del_n, false)
.unwrap();
assert!(deleted.0 > 0);
}
@@ -3032,6 +3050,87 @@ mod tests {
assert_eq!(store.locations[1].volumes_len(), 1);
}
fn add_spread_test_volume(store: &mut Store, vid: u32, collection: &str) {
let spec = VolumeSpec {
collection,
..VolumeSpec::default()
};
store
.add_volume(VolumeId(vid), DiskType::HardDrive, &spec)
.unwrap();
}
fn location_of(store: &Store, vid: u32) -> Option<usize> {
store
.locations
.iter()
.position(|loc| loc.find_volume(VolumeId(vid)).is_some())
}
// Mirrors Go's TestAddVolumeSpreadsCollectionOverDisks.
#[test]
fn test_add_volume_spreads_collection_over_disks() {
let tmp1 = TempDir::new().unwrap();
let tmp2 = TempDir::new().unwrap();
let mut store =
make_test_store(&[tmp1.path().to_str().unwrap(), tmp2.path().to_str().unwrap()]);
add_spread_test_volume(&mut store, 1, "other");
add_spread_test_volume(&mut store, 2, "other");
add_spread_test_volume(&mut store, 3, "spread");
let first = location_of(&store, 3);
// Leave the disk with the "spread" volume holding fewer volumes overall.
let other_on_first = if first == Some(1) { 2 } else { 1 };
store
.delete_volume(VolumeId(other_on_first), false, false, false)
.unwrap();
add_spread_test_volume(&mut store, 4, "spread");
assert_ne!(location_of(&store, 4), first);
}
// Mirrors Go's TestAddVolumeIgnoresFullVolumesOfCollection.
#[test]
fn test_add_volume_ignores_full_volumes_of_collection() {
let tmp1 = TempDir::new().unwrap();
let tmp2 = TempDir::new().unwrap();
let mut store =
make_test_store(&[tmp1.path().to_str().unwrap(), tmp2.path().to_str().unwrap()]);
// Every .dat holds at least its superblock, so all volumes are full.
store.volume_size_limit.store(1, Ordering::Relaxed);
add_spread_test_volume(&mut store, 1, "spread");
let full_disk = location_of(&store, 1);
add_spread_test_volume(&mut store, 2, "other");
add_spread_test_volume(&mut store, 3, "spread");
assert_eq!(location_of(&store, 3), full_disk);
}
// Mirrors Go's TestAddVolumeIgnoresReadOnlyVolumesOfCollection.
#[test]
fn test_add_volume_ignores_read_only_volumes_of_collection() {
let tmp1 = TempDir::new().unwrap();
let tmp2 = TempDir::new().unwrap();
let mut store =
make_test_store(&[tmp1.path().to_str().unwrap(), tmp2.path().to_str().unwrap()]);
add_spread_test_volume(&mut store, 1, "spread");
let writable_disk = location_of(&store, 1);
add_spread_test_volume(&mut store, 2, "other");
add_spread_test_volume(&mut store, 3, "spread");
assert_ne!(location_of(&store, 3), writable_disk);
store
.find_volume(VolumeId(3))
.unwrap()
.1
.set_read_only()
.unwrap();
// The other disk holds more volumes, but its "spread" volume is read-only.
add_spread_test_volume(&mut store, 4, "spread");
assert_ne!(location_of(&store, 4), writable_disk);
}
#[test]
fn test_store_delete_collection() {
let tmp = TempDir::new().unwrap();
@@ -3400,7 +3499,7 @@ mod tests {
cookie: Cookie(2),
..Needle::default()
};
store.delete_volume_needle(vid, &mut del).unwrap();
store.delete_volume_needle(vid, &mut del, false).unwrap();
let size_before = store.find_volume(vid).unwrap().1.dat_file_size().unwrap();
@@ -3724,7 +3823,7 @@ mod tests {
cookie: Cookie(2),
..Needle::default()
};
store.delete_volume_needle(vid, &mut del).unwrap();
store.delete_volume_needle(vid, &mut del, false).unwrap();
let revision = {
let (_, v) = store.find_volume(vid).unwrap();
v.super_block().compaction_revision
@@ -3749,7 +3848,7 @@ mod tests {
cookie: Cookie(4),
..Needle::default()
};
store.delete_volume_needle(vid, &mut del).unwrap();
store.delete_volume_needle(vid, &mut del, false).unwrap();
drop(release_tx);
copy.join().unwrap().unwrap();
+24 -11
View File
@@ -36,10 +36,14 @@ impl Store {
/// `reconcile_ec_shards_across_disks` so the orphan pass can
/// prefer the local idx_directory.
pub fn mirror_ec_metadata_to_shard_disks(&mut self) {
self.mirror_ec_metadata_to_shard_disks_only(None)
}
pub(crate) fn mirror_ec_metadata_to_shard_disks_only(&mut self, only: Option<VolumeId>) {
if self.locations.len() < 2 {
return;
}
let owners = self.index_ecx_owners_for_mirror();
let owners = self.index_ecx_owners_for_mirror(only);
if owners.is_empty() {
return;
}
@@ -54,7 +58,7 @@ impl Store {
}
let mut mirrors: Vec<Mirror> = Vec::new();
for (loc_idx, loc) in self.locations.iter().enumerate() {
let orphans = collect_shard_disk_volumes(loc);
let orphans = collect_shard_disk_volumes(loc, only);
for (key, _shards) in orphans {
let Some(owner) = owners.get(&key) else {
continue;
@@ -113,7 +117,7 @@ impl Store {
// Records both idx_dir (where .ecx was found) and data_dir, so
// the mirror can resolve .vif from data_dir even when .ecx lives
// in idx_directory.
fn index_ecx_owners_for_mirror(&self) -> HashMap<EcKey, EcxOwner> {
fn index_ecx_owners_for_mirror(&self, only: Option<VolumeId>) -> HashMap<EcKey, EcxOwner> {
let mut owners: HashMap<EcKey, EcxOwner> = HashMap::new();
for (loc_idx, loc) in self.locations.iter().enumerate() {
let mut seen: Vec<&str> = Vec::with_capacity(2);
@@ -133,15 +137,18 @@ impl Store {
let Some(base) = name.strip_suffix(".ecx") else {
continue;
};
let Some((collection, vid)) = parse_collection_volume_id_pub(base) else {
continue;
};
if only.is_some_and(|o| o != vid) {
continue;
}
// A 0-byte .ecx is a corrupt stub from a failed copy, not a
// credible owner — skip it so the scan keeps looking for a
// real index on a sibling disk (Go's indexEcxOwners).
if !ent.metadata().is_ok_and(|m| m.len() > 0) {
continue;
}
let Some((collection, vid)) = parse_collection_volume_id_pub(base) else {
continue;
};
owners
.entry(EcKey { collection, vid })
.or_insert_with(|| EcxOwner {
@@ -297,7 +304,10 @@ fn copy_sidecar_atomic(src: &Path, dst: &Path) -> io::Result<()> {
Ok(())
}
fn collect_shard_disk_volumes(loc: &DiskLocation) -> HashMap<EcKey, Vec<String>> {
fn collect_shard_disk_volumes(
loc: &DiskLocation,
only: Option<VolumeId>,
) -> HashMap<EcKey, Vec<String>> {
let mut out: HashMap<EcKey, Vec<String>> = HashMap::new();
let Ok(read) = fs::read_dir(&loc.directory) else {
return out;
@@ -314,13 +324,16 @@ fn collect_shard_disk_volumes(loc: &DiskLocation) -> HashMap<EcKey, Vec<String>>
if crate::storage::disk_location::is_ec_shard_extension(ext).is_none() {
continue;
}
let Some((collection, vid)) = parse_collection_volume_id_pub(base) else {
continue;
};
if only.is_some_and(|o| o != vid) {
continue;
}
match ent.metadata() {
Ok(meta) if meta.len() > 0 => {}
_ => continue,
}
let Some((collection, vid)) = parse_collection_volume_id_pub(base) else {
continue;
};
out.entry(EcKey { collection, vid }).or_default().push(name);
}
out
@@ -688,7 +701,7 @@ mod tests {
add_loc(&mut store, &dir0);
add_loc(&mut store, &dir1);
let owners = store.index_ecx_owners_for_mirror();
let owners = store.index_ecx_owners_for_mirror(None);
let owner = owners
.get(&EcKey {
collection: collection.to_string(),
+115 -18
View File
@@ -92,11 +92,15 @@ impl Store {
///
/// Mirrors `Store.reconcileEcShardsAcrossDisks` in Go.
pub fn reconcile_ec_shards_across_disks(&mut self) {
self.reconcile_ec_shards_across_disks_only(None)
}
pub(crate) fn reconcile_ec_shards_across_disks_only(&mut self, only: Option<VolumeId>) {
if self.locations.len() < 2 {
return;
}
let owners = self.index_ecx_owners();
let owners = self.index_ecx_owners(only);
if owners.is_empty() {
return;
}
@@ -106,7 +110,7 @@ impl Store {
// loc.idx_directory instead of the owner disk.
let mut to_load: Vec<OrphanShardLoad> = Vec::new();
for (loc_idx, loc) in self.locations.iter().enumerate() {
let orphans = collect_orphan_ec_shards(loc, loc_idx);
let orphans = collect_orphan_ec_shards(loc, loc_idx, only);
for (key, shards) in orphans {
let Some(owner) = owners.get(&key) else {
warn!(
@@ -402,7 +406,7 @@ impl Store {
/// matched so downstream loaders point `EcVolume::new` at the
/// directory that really has the file. The first owner found wins;
/// duplicates across disks are unusual but tolerated.
fn index_ecx_owners(&self) -> HashMap<EcKey, EcxOwnerInfo> {
fn index_ecx_owners(&self, only: Option<VolumeId>) -> HashMap<EcKey, EcxOwnerInfo> {
let mut owners: HashMap<EcKey, EcxOwnerInfo> = HashMap::new();
for (loc_idx, loc) in self.locations.iter().enumerate() {
let mut seen: Vec<&str> = Vec::with_capacity(2);
@@ -422,15 +426,18 @@ impl Store {
let Some(base) = name.strip_suffix(".ecx") else {
continue;
};
let Some((collection, vid)) = parse_collection_volume_id_pub(base) else {
continue;
};
if only.is_some_and(|o| o != vid) {
continue;
}
// A 0-byte .ecx is a corrupt stub from a failed copy, not a
// credible owner — skip it so the scan keeps looking for a
// real index on a sibling disk (Go's indexEcxOwners).
if !ent.metadata().is_ok_and(|m| m.len() > 0) {
continue;
}
let Some((collection, vid)) = parse_collection_volume_id_pub(base) else {
continue;
};
let key = EcKey { collection, vid };
owners.entry(key).or_insert(EcxOwnerInfo {
location: loc_idx,
@@ -450,11 +457,18 @@ impl Store {
///
/// Mirrors `Store.CollectEcVolumesMissingIndex` in Go.
pub(crate) fn collect_ec_volumes_missing_index(&self) -> Vec<EcVolumeMissingIndex> {
let owners = self.index_ecx_owners();
self.collect_ec_volumes_missing_index_only(None)
}
pub(crate) fn collect_ec_volumes_missing_index_only(
&self,
only: Option<VolumeId>,
) -> Vec<EcVolumeMissingIndex> {
let owners = self.index_ecx_owners(only);
let mut seen: HashSet<EcKey> = HashSet::new();
let mut missing = Vec::new();
for (loc_idx, loc) in self.locations.iter().enumerate() {
for (key, _shards) in collect_orphan_ec_shards(loc, loc_idx) {
for (key, _shards) in collect_orphan_ec_shards(loc, loc_idx, only) {
if owners.contains_key(&key) || !seen.insert(key.clone()) {
continue;
}
@@ -474,18 +488,24 @@ impl Store {
/// the disks that now have a local index, then fall back to the cross-disk
/// virtual mount. Mirrors `Store.MountRecoveredEcShards` in Go.
pub(crate) fn mount_recovered_ec_shards(&mut self) {
self.mirror_ec_metadata_to_shard_disks();
self.load_orphan_ec_shards_with_local_index();
self.reconcile_ec_shards_across_disks();
self.mount_recovered_ec_shards_only(None)
}
/// `mount_recovered_ec_shards` restricted to one volume's shards when
/// `only` names it.
pub(crate) fn mount_recovered_ec_shards_only(&mut self, only: Option<VolumeId>) {
self.mirror_ec_metadata_to_shard_disks_only(only);
self.load_orphan_ec_shards_with_local_index(only);
self.reconcile_ec_shards_across_disks_only(only);
}
/// Mount on-disk EC shards whose `.ecx` index is now present on the same disk.
/// Unlike `reconcile_ec_shards_across_disks` it needs no sibling disk, so a
/// single-disk store recovers once its index has been fetched from a peer.
fn load_orphan_ec_shards_with_local_index(&mut self) {
fn load_orphan_ec_shards_with_local_index(&mut self, only: Option<VolumeId>) {
let mut work: Vec<(usize, EcKey, Vec<ShardId>)> = Vec::new();
for (loc_idx, loc) in self.locations.iter().enumerate() {
for (key, shards) in collect_orphan_ec_shards(loc, loc_idx) {
for (key, shards) in collect_orphan_ec_shards(loc, loc_idx, only) {
if !loc.has_ecx_file_on_disk(&key.collection, key.vid) {
continue;
}
@@ -516,6 +536,7 @@ impl Store {
fn collect_orphan_ec_shards(
loc: &crate::storage::disk_location::DiskLocation,
_loc_idx: usize,
only: Option<VolumeId>,
) -> HashMap<EcKey, Vec<(String, ShardId)>> {
let mut orphans: HashMap<EcKey, Vec<(String, ShardId)>> = HashMap::new();
let Ok(read) = fs::read_dir(&loc.directory) else {
@@ -533,6 +554,12 @@ fn collect_orphan_ec_shards(
let Some(shard_id) = is_ec_shard_extension(ext) else {
continue;
};
let Some((collection, vid)) = parse_collection_volume_id_pub(base) else {
continue;
};
if only.is_some_and(|o| o != vid) {
continue;
}
// Ignore zero-byte shards. Use the DirEntry's metadata so we
// don't pay a second stat syscall per file beyond what
// read_dir already returned.
@@ -540,9 +567,6 @@ fn collect_orphan_ec_shards(
Ok(meta) if meta.len() > 0 => {}
_ => continue,
}
let Some((collection, vid)) = parse_collection_volume_id_pub(base) else {
continue;
};
// Skip shards that are already registered to an EcVolume.
if let Some(ecv) = loc.find_ec_volume(vid)
&& ecv.has_shard(shard_id)
@@ -655,7 +679,7 @@ mod tests {
std::fs::write(ec_local_ecx_path(&d0, "pics", VolumeId(7)), b"").unwrap();
write_index_files(&d1, "pics", 7, 10, 4);
let owners = store.index_ecx_owners();
let owners = store.index_ecx_owners(None);
let owner = owners
.get(&EcKey {
collection: "pics".to_string(),
@@ -667,7 +691,7 @@ mod tests {
// A stub with no real index anywhere owns nothing.
std::fs::write(ec_local_ecx_path(&d0, "pics", VolumeId(8)), b"").unwrap();
assert!(!store.index_ecx_owners().contains_key(&EcKey {
assert!(!store.index_ecx_owners(None).contains_key(&EcKey {
collection: "pics".to_string(),
vid: VolumeId(8),
}));
@@ -1240,6 +1264,79 @@ mod tests {
);
}
#[test]
fn test_mount_recovered_ec_shards_only_named_volume() {
// Recovering one volume mounts only its shards. Another volume's orphan
// shards, such as ones an in-flight copy just renamed in, stay unmounted
// even when an index for them is already on disk. Mirrors Go
// TestMountRecoveredEcShards_OnlyNamedVolume.
let tmp = TempDir::new().unwrap();
let dir0 = tmp.path().join("data0");
let dir1 = tmp.path().join("data1");
std::fs::create_dir_all(&dir0).unwrap();
std::fs::create_dir_all(&dir1).unwrap();
let collection = "col";
let (recovered_vid, other_vid) = (71u32, 72u32);
let mut store = Store::new(NeedleMapKind::InMemory);
for d in [&dir0, &dir1] {
store
.add_location(
d.to_str().unwrap(),
d.to_str().unwrap(),
100,
DiskType::HardDrive,
MinFreeSpace::Percent(0.0),
Vec::new(),
)
.unwrap();
}
for vid in [recovered_vid, other_vid] {
write_shard(dir0.to_str().unwrap(), collection, vid, 0);
write_shard(dir1.to_str().unwrap(), collection, vid, 1);
}
write_index_files(dir0.to_str().unwrap(), collection, other_vid, 10, 4);
let missing = store.collect_ec_volumes_missing_index();
let m = missing
.iter()
.find(|m| m.vid == VolumeId(recovered_vid))
.expect("recovered volume not reported as missing its index");
write_index_files(&m.idx_dir, collection, recovered_vid, 10, 4);
store.mount_recovered_ec_shards_only(Some(VolumeId(recovered_vid)));
for (i, loc) in store.locations.iter().enumerate() {
let ev = loc
.find_ec_volume(VolumeId(recovered_vid))
.unwrap_or_else(|| panic!("volume {} not mounted on disk{}", recovered_vid, i));
assert!(
ev.has_shard(i as u8),
"shard {}.{} not mounted",
recovered_vid,
i
);
assert!(
loc.find_ec_volume(VolumeId(other_vid)).is_none(),
"volume {} mounted on disk{} by the recovery of volume {}",
other_vid,
i,
recovered_vid,
);
}
let mirrored = format!(
"{}/{}_{}.ecx",
dir1.to_str().unwrap(),
collection,
other_vid
);
assert!(
!std::path::Path::new(&mirrored).exists(),
"volume {} index mirrored to disk1 by the recovery of volume {}",
other_vid,
recovered_vid,
);
}
#[test]
fn test_collect_missing_index_excludes_cross_disk_orphan() {
// A .ecx merely on a sibling disk is the same-server case the reconcile
File diff suppressed because it is too large. Load diff
@@ -188,7 +188,7 @@ mod tests {
for id in 1..=3 {
v.write_needle(&mut needle(id), true, false).unwrap();
}
v.delete_needle(&mut needle(2)).unwrap();
v.delete_needle(&mut needle(2), false).unwrap();
v.sync_to_disk().unwrap();
let (want_count, want_deleted) = (v.file_count(), v.deleted_count());
drop(v);
@@ -382,10 +382,13 @@ pub(crate) mod tests {
write_test_volume(dir, 6);
let v = open_volume(dir);
v.delete_needle(&mut Needle {
id: NeedleId(3),
..Needle::default()
})
v.delete_needle(
&mut Needle {
id: NeedleId(3),
..Needle::default()
},
false,
)
.unwrap();
drop(v);
@@ -417,10 +420,13 @@ pub(crate) mod tests {
write_test_volume(dir, 8);
let v = open_volume(dir);
v.delete_needle(&mut Needle {
id: NeedleId(2),
..Needle::default()
})
v.delete_needle(
&mut Needle {
id: NeedleId(2),
..Needle::default()
},
false,
)
.unwrap();
drop(v);
@@ -505,10 +505,13 @@ mod tests {
f.written.insert(*id, n.data.clone());
}
Op::Del(id) => {
v.delete_needle(&mut Needle {
id: NeedleId(*id),
..Needle::default()
})
v.delete_needle(
&mut Needle {
id: NeedleId(*id),
..Needle::default()
},
false,
)
.unwrap();
}
Op::Empty(id) => {
+118
View File
@@ -110,6 +110,7 @@ fn build_test_state(
vacuum_commit_slots: None,
vacuum_commits_waiting: std::sync::atomic::AtomicU32::new(0),
is_heartbeating: std::sync::atomic::AtomicBool::new(true),
ec_lifecycle_locks: std::sync::Mutex::new(std::collections::HashMap::new()),
ec_decodes_in_flight: std::sync::Mutex::new(std::collections::HashSet::new()),
volume_copies_in_flight: std::sync::Arc::new(std::sync::Mutex::new(std::collections::HashSet::new())),
ec_decode_tail: std::sync::Mutex::new(std::collections::HashSet::new()),
@@ -1435,3 +1436,120 @@ async fn write_queue_upload_returns_same_etag_as_direct_write() {
}
assert_eq!(etags[1], etags[0], "unchanged upload ETag differs");
}
// ============================================================================
// A Range over a compressed needle is answered from the decoded bytes
// ============================================================================
#[tokio::test]
async fn range_with_accept_gzip_on_compressed_needle_serves_decoded_bytes() {
let (state, _tmp) = test_state();
let uri = "/1/03637037d6/notes.txt";
let plain: Vec<u8> = (0..400)
.flat_map(|i| format!("line {i:06}: the quick brown fox\n").into_bytes())
.collect();
let app = build_admin_router(state.clone());
let response = app
.oneshot(
Request::builder()
.method("POST")
.uri(uri)
.body(Body::from(plain.clone()))
.unwrap(),
)
.await
.unwrap();
assert_eq!(response.status(), StatusCode::CREATED);
let read = |range: Option<&str>| {
let mut request = Request::builder()
.uri(uri)
.header("Accept-Encoding", "gzip");
if let Some(range) = range {
request = request.header("Range", range);
}
build_admin_router(state.clone()).oneshot(request.body(Body::empty()).unwrap())
};
// The whole read passes the stored gzip through, so its length is the compressed size.
let response = read(None).await.unwrap();
assert_eq!(response.status(), StatusCode::OK);
assert_eq!(response.headers()["Content-Encoding"], "gzip");
let compressed_len = body_bytes(response).await.len();
assert!(compressed_len < plain.len() / 4);
for (start, end) in [(10, 69), (compressed_len + 10, compressed_len + 69)] {
let response = read(Some(&format!("bytes={start}-{end}"))).await.unwrap();
assert_eq!(response.status(), StatusCode::PARTIAL_CONTENT);
assert!(response.headers().get("Content-Encoding").is_none());
assert_eq!(
response.headers()["Content-Range"],
format!("bytes {start}-{end}/{}", plain.len()).as_str()
);
assert_eq!(body_bytes(response).await, &plain[start..=end]);
}
}
// ============================================================================
// HEAD of a compressed needle reports the length the matching GET would send
// ============================================================================
#[tokio::test]
async fn head_on_compressed_needle_reports_served_length() {
let (state, _tmp) = test_state();
let uri = "/1/04637037d6/notes.txt";
let plain: Vec<u8> = (0..400)
.flat_map(|i| format!("line {i:06}: the quick brown fox\n").into_bytes())
.collect();
let app = build_admin_router(state.clone());
let response = app
.oneshot(
Request::builder()
.method("POST")
.uri(uri)
.body(Body::from(plain.clone()))
.unwrap(),
)
.await
.unwrap();
assert_eq!(response.status(), StatusCode::CREATED);
let send = |method: &str, accept_gzip: bool, range: Option<&str>| {
let mut request = Request::builder().method(method).uri(uri);
if accept_gzip {
request = request.header("Accept-Encoding", "gzip");
}
if let Some(range) = range {
request = request.header("Range", range);
}
build_admin_router(state.clone()).oneshot(request.body(Body::empty()).unwrap())
};
let response = send("GET", true, None).await.unwrap();
let compressed_len = body_bytes(response).await.len();
assert!(compressed_len < plain.len() / 4);
for (accept_gzip, range, encoding, length) in [
(false, None, None, plain.len()),
(true, None, Some("gzip"), compressed_len),
(true, Some("bytes=0-9"), None, plain.len()),
] {
let response = send("HEAD", accept_gzip, range).await.unwrap();
assert_eq!(response.status(), StatusCode::OK);
assert_eq!(
response
.headers()
.get("Content-Encoding")
.map(|v| v.to_str().unwrap()),
encoding,
"accept_gzip={accept_gzip} range={range:?}"
);
assert_eq!(
response.headers()["Content-Length"],
length.to_string().as_str(),
"accept_gzip={accept_gzip} range={range:?}"
);
}
}
+4
View File
@@ -7157,6 +7157,7 @@ dependencies = [
"arrow-array",
"arrow-cast",
"arrow-schema",
"arrow-select",
"async-trait",
"base64 0.22.1",
"bytes",
@@ -7170,8 +7171,10 @@ dependencies = [
"hmac 0.12.1",
"http 1.5.0",
"hyper-util",
"jsonb",
"jsonwebtoken",
"lance",
"lance-arrow",
"lance-datafusion",
"lance-index",
"lance-linalg",
@@ -7203,6 +7206,7 @@ dependencies = [
"tower",
"tracing",
"tracing-subscriber",
"uuid",
"x509-parser",
"zstd",
]
+39 -2
View File
@@ -3,7 +3,7 @@ use tokio::sync::mpsc;
use crate::pb::{
ActivityEvent, DetectionComplete, DetectionProposals, JobCompleted, JobProgressUpdate,
WorkerObservations, WorkerToAdminMessage, worker_to_admin_message::Body,
JobState, WorkerObservations, WorkerToAdminMessage, worker_to_admin_message::Body,
};
/// Replies to one detection request.
@@ -65,7 +65,12 @@ impl DetectionSender for StreamSender {
}
impl ExecutionSender for StreamSender {
fn send_progress(&self, progress: JobProgressUpdate) -> Result<()> {
fn send_progress(&self, mut progress: JobProgressUpdate) -> Result<()> {
// the Go handlers always report a state; older admins read a missing
// one as an ended job and drop its completion
if progress.state == JobState::Unspecified as i32 {
progress.state = JobState::Running as i32;
}
self.send(Body::JobProgressUpdate(progress))
}
@@ -200,4 +205,36 @@ mod tests {
.expect("send");
assert!(metered.reported_failure());
}
#[test]
fn progress_without_a_state_is_sent_as_running() {
let (tx, mut rx) = mpsc::unbounded_channel();
let stream = StreamSender::new("worker-1".to_string(), tx);
let metered = MeteredSender::new(&stream);
metered
.send_progress(JobProgressUpdate {
progress_percent: 10.0,
..Default::default()
})
.expect("send");
metered
.send_progress(JobProgressUpdate {
state: JobState::Failed as i32,
..Default::default()
})
.expect("send");
let mut states = Vec::new();
while let Ok(message) = rx.try_recv() {
let Some(Body::JobProgressUpdate(progress)) = message.body else {
panic!("expected a progress update");
};
states.push(progress.state);
}
assert_eq!(
states,
vec![JobState::Running as i32, JobState::Failed as i32]
);
}
}
+8 -1
View File
@@ -35,10 +35,16 @@ lance-table = "10"
# For the execution options the sort needs; the scanner's own stream helper
# takes the default, which has spilling off.
lance-datafusion = "10"
arrow-array = "58"
arrow-schema = "58"
arrow-cast = "58"
arrow-select = "58"
# Lance's stored form of the metadata JSON, read when deriving the
# indexed-key columns.
jsonb = { version = "0.5", default-features = false }
chrono = "0.4"
futures = "0.3"
uuid = "1"
anyhow.workspace = true
async-trait.workspace = true
clap = { version = "4", features = ["derive", "env"] }
@@ -88,9 +94,10 @@ protoc-bin-vendored = "3"
[dev-dependencies]
tokio = { workspace = true, features = ["test-util"] }
tonic.workspace = true
arrow-array = "58"
lance-arrow = "10"
arrow-schema = "58"
arrow-cast = "58"
arrow-select = "58"
lance-linalg = "10"
tempfile = "3"
+18 -2
View File
@@ -18,7 +18,8 @@ use lance::io::{
use lance::session::Session;
use crate::catalog::{NamespaceClient, TableDescription};
use crate::direct_read::DirectReads;
use crate::direct_read::{DirectReads, Wrappers};
use crate::drop_guard::DropGuard;
/// A table the worker is about to work on.
pub struct OpenTable {
@@ -75,7 +76,7 @@ pub fn unsigned_options() -> HashMap<String, String> {
pub async fn open(client: &NamespaceClient, id: &[String], storage: &Storage) -> Result<OpenTable> {
let description = client.describe_table(id).await?;
let wrapper = storage.wrapper(&description.location).await?;
let dataset = open_at(&description, wrapper, None).await?;
let dataset = open_guarded(&description, wrapper).await?;
Ok(OpenTable {
id: id.to_vec(),
location: description.location,
@@ -83,6 +84,21 @@ pub async fn open(client: &NamespaceClient, id: &[String], storage: &Storage) ->
})
}
/// [`open_at`] for a job that writes the table: its writes fail once the
/// table is dropped under it (see [`DropGuard`]).
pub async fn open_guarded(
description: &TableDescription,
wrapper: Option<Arc<dyn WrappingObjectStore>>,
) -> Result<Dataset> {
let guard = DropGuard::new();
let mut wrappers: Vec<Arc<dyn WrappingObjectStore>> = wrapper.into_iter().collect();
wrappers.push(guard.clone());
let dataset = open_at(description, Some(Arc::new(Wrappers(wrappers))), None).await?;
let opened = dataset.manifest_location();
guard.arm(opened.path.clone(), opened.e_tag.clone());
Ok(dataset)
}
/// Open a table the caller has already resolved, with the store wrapped, in
/// `session` when given, otherwise in a session of its own.
pub async fn open_at(
@@ -0,0 +1,322 @@
//! Keeps a job from writing into a table dropped under it.
//!
//! DeleteIndex purges a table's directory, and an index created again under
//! the same name reuses it. A job that opened the table before the purge
//! would otherwise go on writing there: its data files would land in the
//! new table's directory, and its commit would put a manifest of the old
//! table there, so the new table could not be created, or would read it.
use std::fmt;
use std::ops::Range;
use std::sync::{Arc, Mutex};
use bytes::Bytes;
use futures::StreamExt;
use futures::stream::BoxStream;
use lance::io::WrappingObjectStore;
use object_store::path::Path;
use object_store::{
CopyOptions, GetOptions, GetResult, ListResult, MultipartUpload, ObjectMeta, ObjectStore,
ObjectStoreExt, PutMultipartOptions, PutOptions, PutPayload, PutResult, RenameOptions, Result,
UploadPart,
};
/// Fails every write once the manifest it watches is gone or is another
/// object (a table created again reuses version paths): the one of the
/// version the job opened, then the newest the job committed. Armed after the
/// open; until then it lets everything through. When the job itself deletes
/// the watched manifest, removing old versions as a purge does, it watches
/// the newest manifest left instead.
#[derive(Debug, Default)]
pub struct DropGuard {
watched: Arc<Mutex<Watched>>,
}
#[derive(Debug, Default, Clone)]
struct Watched {
manifest: Option<Path>,
/// The manifest's e-tag when known, telling it from a replacement.
e_tag: Option<String>,
/// The job deleted the watched manifest itself.
deleted: bool,
}
impl DropGuard {
pub fn new() -> Arc<Self> {
Arc::new(Self::default())
}
/// Watch `manifest`, the opened version's, with its e-tag when known.
pub fn arm(&self, manifest: Path, e_tag: Option<String>) {
let mut watched = self.watched.lock().unwrap();
watched.manifest = Some(manifest);
watched.e_tag = e_tag;
}
}
impl WrappingObjectStore for DropGuard {
fn wrap(&self, _prefix: &str, original: Arc<dyn ObjectStore>) -> Arc<dyn ObjectStore> {
Arc::new(Guarded {
inner: original,
watched: self.watched.clone(),
})
}
}
#[derive(Debug, Clone)]
struct Guarded {
inner: Arc<dyn ObjectStore>,
watched: Arc<Mutex<Watched>>,
}
/// What a check found: the table is there, or it was dropped, or created
/// again, with the error to fail the write with.
enum Found {
Table,
Dropped(object_store::Error),
Replaced(object_store::Error),
}
impl Guarded {
/// Whether the watched manifest is still the table's. An error is a
/// failed probe, which says nothing about the table.
async fn find(&self, location: &Path) -> Result<Found> {
let watched = self.watched.lock().unwrap().clone();
let Some(mut manifest) = watched.manifest else {
return Ok(Found::Table);
};
let mut e_tag = watched.e_tag;
if watched.deleted {
if let Some((newest, tag)) = self.newest_manifest(&manifest).await? {
manifest = newest;
e_tag = tag;
}
let mut watched = self.watched.lock().unwrap();
watched.manifest = Some(manifest.clone());
watched.e_tag = e_tag.clone();
watched.deleted = false;
}
let error = |why: &str| {
object_store::Error::Generic {
store: "DropGuard",
source: format!(
"the table was dropped while this job wrote it: {manifest} {why}, so {location} is not written"
)
.into(),
}
};
match self.inner.head(&manifest).await {
Ok(meta) => match (&e_tag, &meta.e_tag) {
(Some(want), Some(have)) if want != have => {
Ok(Found::Replaced(error("is another table's")))
}
_ => Ok(Found::Table),
},
// Gone with manifests beside it: a table created again, at an
// earlier version than the one watched.
Err(object_store::Error::NotFound { .. }) => {
match self.newest_manifest(&manifest).await? {
Some(_) => Ok(Found::Replaced(error("is gone and another table is there"))),
None => Ok(Found::Dropped(error("is gone"))),
}
}
Err(e) => Err(e),
}
}
async fn check(&self, location: &Path) -> Result<()> {
match self.find(location).await? {
Found::Table => Ok(()),
Found::Dropped(error) | Found::Replaced(error) => Err(error),
}
}
}
fn is_manifest(location: &Path) -> bool {
location.extension() == Some("manifest")
}
impl Guarded {
/// The most recently written manifest beside `manifest`, with its e-tag.
async fn newest_manifest(&self, manifest: &Path) -> Result<Option<(Path, Option<String>)>> {
let dir: Path = manifest
.parts()
.take(manifest.parts().count() - 1)
.collect();
let listed = self.inner.list_with_delimiter(Some(&dir)).await?;
Ok(listed
.objects
.into_iter()
.filter(|o| is_manifest(&o.location))
.max_by_key(|o| o.last_modified)
.map(|o| (o.location, o.e_tag)))
}
fn committed(&self, location: &Path, e_tag: Option<String>) {
if is_manifest(location) {
let mut watched = self.watched.lock().unwrap();
if watched.manifest.is_some() {
watched.manifest = Some(location.clone());
watched.e_tag = e_tag;
watched.deleted = false;
}
}
}
}
/// A multipart upload that checks the table again before it completes.
#[derive(Debug)]
struct GuardedUpload {
inner: Box<dyn MultipartUpload>,
guard: Guarded,
location: Path,
}
#[async_trait::async_trait]
impl MultipartUpload for GuardedUpload {
fn put_part(&mut self, data: PutPayload) -> UploadPart {
self.inner.put_part(data)
}
async fn complete(&mut self) -> Result<PutResult> {
// A failed probe aborts too: the parts would outlive the upload.
let error = match self.guard.find(&self.location).await {
Ok(Found::Table) => return self.inner.complete().await,
Ok(Found::Dropped(error) | Found::Replaced(error)) | Err(error) => error,
};
let _ = self.inner.abort().await;
Err(error)
}
async fn abort(&mut self) -> Result<()> {
self.inner.abort().await
}
}
impl fmt::Display for Guarded {
fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
write!(f, "DropGuard({})", self.inner)
}
}
#[async_trait::async_trait]
impl ObjectStore for Guarded {
async fn put_opts(
&self,
location: &Path,
payload: PutPayload,
opts: PutOptions,
) -> Result<PutResult> {
self.check(location).await?;
let put = self.inner.put_opts(location, payload, opts).await?;
// A drop can land between the check and the write. A manifest is
// what makes the table readable again, so look once more and take
// it back if the table is gone; a failed probe leaves it. A table
// created again since may have written its own manifest there:
// only a manifest known to be this write's is taken back.
if is_manifest(location) {
let (error, replaced) = match self.find(location).await {
Ok(Found::Dropped(error)) => (error, false),
Ok(Found::Replaced(error)) => (error, true),
_ => {
self.committed(location, put.e_tag.clone());
return Ok(put);
}
};
let ours = match (&put.e_tag, self.inner.head(location).await) {
(Some(wrote), Ok(meta)) => meta.e_tag.as_ref() == Some(wrote),
(_, Err(object_store::Error::NotFound { .. })) => false,
_ => !replaced,
};
if ours {
let _ = self.inner.delete(location).await;
}
return Err(error);
}
self.committed(location, put.e_tag.clone());
Ok(put)
}
async fn put_multipart_opts(
&self,
location: &Path,
opts: PutMultipartOptions,
) -> Result<Box<dyn MultipartUpload>> {
self.check(location).await?;
let inner = self.inner.put_multipart_opts(location, opts).await?;
Ok(Box::new(GuardedUpload {
inner,
guard: self.clone(),
location: location.clone(),
}))
}
async fn get_opts(&self, location: &Path, options: GetOptions) -> Result<GetResult> {
self.inner.get_opts(location, options).await
}
async fn get_ranges(&self, location: &Path, ranges: &[Range<u64>]) -> Result<Vec<Bytes>> {
self.inner.get_ranges(location, ranges).await
}
fn delete_stream(
&self,
locations: BoxStream<'static, Result<Path>>,
) -> BoxStream<'static, Result<Path>> {
// A cleanup picks its paths before it deletes them; a table created
// again meanwhile reuses those paths, so look once before deleting.
// In a dropped table there is nothing left to spare, and the job's
// own failed commit is cleaned up.
let watched = self.watched.clone();
let locations = locations.inspect(move |location| {
if let Ok(location) = location {
let mut watched = watched.lock().unwrap();
if watched.manifest.as_ref() == Some(location) {
watched.deleted = true;
}
}
});
let guard = self.clone();
let pending = Arc::new(Mutex::new(Some(locations.boxed())));
let checked =
futures::stream::once(async move { guard.find(&Path::from("cleanup")).await })
.flat_map(move |found| {
let locations = pending.lock().unwrap().take();
match (found, locations) {
(Ok(Found::Table | Found::Dropped(_)), Some(locations)) => locations,
(Ok(Found::Replaced(error)), _) | (Err(error), _) => {
futures::stream::once(async move { Err(error) }).boxed()
}
(Ok(_), None) => futures::stream::empty().boxed(),
}
})
.boxed();
self.inner.delete_stream(checked)
}
fn list(&self, prefix: Option<&Path>) -> BoxStream<'static, Result<ObjectMeta>> {
self.inner.list(prefix)
}
fn list_with_offset(
&self,
prefix: Option<&Path>,
offset: &Path,
) -> BoxStream<'static, Result<ObjectMeta>> {
self.inner.list_with_offset(prefix, offset)
}
async fn list_with_delimiter(&self, prefix: Option<&Path>) -> Result<ListResult> {
self.inner.list_with_delimiter(prefix).await
}
async fn copy_opts(&self, from: &Path, to: &Path, options: CopyOptions) -> Result<()> {
self.check(to).await?;
self.inner.copy_opts(from, to, options).await
}
async fn rename_opts(&self, from: &Path, to: &Path, options: RenameOptions) -> Result<()> {
self.check(to).await?;
self.inner.rename_opts(from, to, options).await
}
}
@@ -0,0 +1,701 @@
//! A table dropped while a worker job runs on it: the job must fail without
//! writing into the dropped path, and a table created again there starts
//! empty.
use std::fmt;
use std::ops::Range;
use std::path::PathBuf;
use std::sync::Arc;
use std::sync::atomic::{AtomicBool, AtomicUsize, Ordering};
use bytes::Bytes;
use futures::StreamExt;
use futures::stream::BoxStream;
use lance::Dataset;
use lance::io::WrappingObjectStore;
use object_store::path::Path;
use object_store::{
CopyOptions, GetOptions, GetResult, ListResult, MultipartUpload, ObjectMeta, ObjectStore,
ObjectStoreExt, PutMultipartOptions, PutOptions, PutPayload, PutResult, RenameOptions, Result,
};
use crate::metadata_index::tests::lifecycle::{table, upsert};
use crate::metadata_index::{self, State};
/// Removes the table directory just before the first write through it, the
/// way DeleteIndex purging the table between a job's open and its writes
/// would.
#[derive(Debug)]
struct DropOnFirstWrite {
dir: PathBuf,
fired: Arc<AtomicBool>,
}
impl WrappingObjectStore for DropOnFirstWrite {
fn wrap(&self, _prefix: &str, original: Arc<dyn ObjectStore>) -> Arc<dyn ObjectStore> {
Arc::new(Dropping {
inner: original,
dir: self.dir.clone(),
fired: self.fired.clone(),
})
}
}
#[derive(Debug)]
struct Dropping {
inner: Arc<dyn ObjectStore>,
dir: PathBuf,
fired: Arc<AtomicBool>,
}
impl Dropping {
fn drop_once(&self) {
if !self.fired.swap(true, Ordering::SeqCst) {
std::fs::remove_dir_all(&self.dir).unwrap();
}
}
}
impl fmt::Display for Dropping {
fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
write!(f, "Dropping({})", self.inner)
}
}
#[async_trait::async_trait]
impl ObjectStore for Dropping {
async fn put_opts(
&self,
location: &Path,
payload: PutPayload,
opts: PutOptions,
) -> Result<PutResult> {
self.drop_once();
self.inner.put_opts(location, payload, opts).await
}
async fn put_multipart_opts(
&self,
location: &Path,
opts: PutMultipartOptions,
) -> Result<Box<dyn MultipartUpload>> {
self.drop_once();
self.inner.put_multipart_opts(location, opts).await
}
async fn get_opts(&self, location: &Path, options: GetOptions) -> Result<GetResult> {
self.inner.get_opts(location, options).await
}
async fn get_ranges(&self, location: &Path, ranges: &[Range<u64>]) -> Result<Vec<Bytes>> {
self.inner.get_ranges(location, ranges).await
}
fn delete_stream(
&self,
locations: BoxStream<'static, Result<Path>>,
) -> BoxStream<'static, Result<Path>> {
self.inner.delete_stream(locations)
}
fn list(&self, prefix: Option<&Path>) -> BoxStream<'static, Result<ObjectMeta>> {
self.inner.list(prefix)
}
fn list_with_offset(
&self,
prefix: Option<&Path>,
offset: &Path,
) -> BoxStream<'static, Result<ObjectMeta>> {
self.inner.list_with_offset(prefix, offset)
}
async fn list_with_delimiter(&self, prefix: Option<&Path>) -> Result<ListResult> {
self.inner.list_with_delimiter(prefix).await
}
async fn copy_opts(&self, from: &Path, to: &Path, options: CopyOptions) -> Result<()> {
self.inner.copy_opts(from, to, options).await
}
async fn rename_opts(&self, from: &Path, to: &Path, options: RenameOptions) -> Result<()> {
self.inner.rename_opts(from, to, options).await
}
}
/// Opens the table the way a job does; `during` drops it at the job's first
/// write, otherwise it is dropped right after the open.
async fn open_then_drop(dir: &tempfile::TempDir, during: bool) -> Dataset {
let description = crate::catalog::TableDescription {
location: dir.path().to_str().unwrap().to_string(),
storage_options: Default::default(),
managed_versioning: false,
};
let dropping: Option<Arc<dyn WrappingObjectStore>> = during.then(|| {
Arc::new(DropOnFirstWrite {
dir: dir.path().to_path_buf(),
fired: Arc::new(AtomicBool::new(false)),
}) as Arc<dyn WrappingObjectStore>
});
let dataset = crate::dataset::open_guarded(&description, dropping)
.await
.unwrap();
if !during {
std::fs::remove_dir_all(dir.path()).unwrap();
}
dataset
}
fn files(dir: &std::path::Path) -> Vec<String> {
let mut out = Vec::new();
if let Ok(entries) = std::fs::read_dir(dir) {
for entry in entries.flatten() {
let path = entry.path();
if path.is_dir() {
out.extend(files(&path));
} else {
out.push(path.display().to_string());
}
}
}
out
}
/// Runs `job` on a table dropped right after the job opens it, or at the
/// job's first write. The job must fail, leave no manifest (nor, dropped
/// before it wrote, any file) in the dropped path, and a table created
/// there again must start empty.
async fn dropped_during<F, Fut>(
label: &str,
during: bool,
prepare: impl AsyncFnOnce(&mut Dataset),
job: F,
) where
F: FnOnce(Dataset) -> Fut,
Fut: std::future::Future<Output = anyhow::Result<()>>,
{
let dir = tempfile::TempDir::new().unwrap();
let mut dataset = table(&dir, 4).await;
prepare(&mut dataset).await;
let result = job(open_then_drop(&dir, during).await).await;
let left = files(dir.path());
let error = result.expect_err(&format!(
"{label}: a job on a dropped table succeeded, leaving {left:?}"
));
eprintln!(
"{label} (dropped {}): {error:#}",
if during {
"at its first write"
} else {
"after the open"
}
);
assert!(
!left.iter().any(|f| f.ends_with(".manifest")),
"{label}: a manifest was written into the dropped table: {left:?}"
);
if !during {
assert!(
left.is_empty(),
"{label}: files were written into the dropped table: {left:?}"
);
}
let recreated = Dataset::write(
arrow_array::RecordBatchIterator::new(
Vec::<std::result::Result<arrow_array::RecordBatch, arrow_schema::ArrowError>>::new(),
Arc::new(crate::metadata_index::tests::lifecycle::base_schema()),
),
dir.path().to_str().unwrap(),
None,
)
.await
.unwrap_or_else(|e| panic!("{label}: the table could not be created again: {e}"));
assert_eq!(recreated.version().version, 1, "{label}");
assert_eq!(recreated.count_rows(None).await.unwrap(), 0, "{label}");
}
/// Every job, with the table dropped after its open and at its first write.
#[tokio::test(flavor = "multi_thread", worker_threads = 4)]
async fn a_job_on_a_dropped_table_fails_and_leaves_it_dropped() {
for during in [false, true] {
dropped_during(
"vector index build",
during,
async |_| {},
async |mut d| {
crate::vector_index::build(
&mut d,
"vector",
crate::vector_index::INDEX_NAME,
4,
false,
)
.await
},
)
.await;
dropped_during(
"metadata key add",
during,
async |d| metadata_index::request(d, "author").await.unwrap(),
async |mut d| {
let key = metadata_index::add_column(&mut d, "author").await?;
metadata_index::finish_after(&mut d, &key, 1_000, std::time::Duration::ZERO)
.await
.map(|_| ())
},
)
.await;
dropped_during(
"metadata key maintenance",
during,
async |d| {
let key = metadata_index::add_column(d, "author").await.unwrap();
metadata_index::finish_after(d, &key, 1_000, std::time::Duration::ZERO)
.await
.unwrap();
*d = upsert(d.clone(), 4_000..4_500).await;
},
async |mut d| {
let key = metadata_index::keys_of(&d).remove(0);
metadata_index::maintain(&mut d, &key, 1_000, 0)
.await
.map(|_| ())
},
)
.await;
dropped_during(
"metadata key remove",
during,
async |d| {
let key = metadata_index::add_column(d, "author").await.unwrap();
metadata_index::finish_after(d, &key, 1_000, std::time::Duration::ZERO)
.await
.unwrap();
metadata_index::set_state(d, "_mk_0", State::Off)
.await
.unwrap();
},
async |mut d| metadata_index::remove_column(&mut d, "_mk_0").await,
)
.await;
}
}
/// A job that removes old versions itself, as a purge does, removes the
/// manifest it opened: that is no drop, and its later writes go through.
#[tokio::test(flavor = "multi_thread", worker_threads = 4)]
async fn a_purge_through_the_guard_completes() {
let dir = tempfile::TempDir::new().unwrap();
let dataset = table(&dir, 4).await;
let mut dataset = upsert(dataset, 0..100).await;
dataset.delete("key = 'k1' OR key = 'k2001'").await.unwrap();
let description = crate::catalog::TableDescription {
location: dir.path().to_str().unwrap().to_string(),
storage_options: Default::default(),
managed_versioning: false,
};
let mut guarded = crate::dataset::open_guarded(&description, None)
.await
.unwrap();
// The job commits, then removes every older version, the one it opened
// with them.
guarded
.update_config([("seaweedfs.test", "1")])
.await
.unwrap();
let removed = lance::dataset::cleanup::cleanup_old_versions(
&guarded,
lance::dataset::cleanup::CleanupPolicy {
before_timestamp: Some(chrono::Utc::now() + chrono::Duration::hours(1)),
before_version: Some(guarded.version().version),
..Default::default()
},
)
.await
.unwrap();
assert!(removed.old_versions > 0);
guarded
.update_config([("seaweedfs.test", "2")])
.await
.unwrap();
crate::jobs::purge::purge(&mut guarded, "t1").await.unwrap();
assert_eq!(guarded.count_rows(None).await.unwrap(), 3_998);
}
/// The table is dropped and created again up to the version the job opened,
/// so the manifest path the job watches exists again, as another object:
/// the job must still fail and leave the new table as it was.
#[tokio::test(flavor = "multi_thread", worker_threads = 4)]
async fn a_job_on_a_table_created_again_fails() {
let dir = tempfile::TempDir::new().unwrap();
let mut dataset = table(&dir, 4).await;
let key = metadata_index::add_column(&mut dataset, "author")
.await
.unwrap();
metadata_index::finish_after(&mut dataset, &key, 1_000, std::time::Duration::ZERO)
.await
.unwrap();
metadata_index::set_state(&mut dataset, "_mk_0", State::Off)
.await
.unwrap();
let description = crate::catalog::TableDescription {
location: dir.path().to_str().unwrap().to_string(),
storage_options: Default::default(),
managed_versioning: false,
};
let mut job = crate::dataset::open_guarded(&description, None)
.await
.unwrap();
let opened = job.version().version;
std::fs::remove_dir_all(dir.path()).unwrap();
let mut recreated = table(&dir, 1).await;
while recreated.version().version < opened {
let version = recreated.version().version.to_string();
recreated
.update_config([("seaweedfs.test", version.as_str())])
.await
.unwrap();
}
assert_eq!(recreated.version().version, opened);
let error = metadata_index::remove_column(&mut job, "_mk_0")
.await
.expect_err("a job on a table created again succeeded");
assert!(format!("{error:#}").contains("DropGuard"), "{error:#}");
recreated.checkout_latest().await.unwrap();
assert_eq!(recreated.version().version, opened);
assert_eq!(recreated.count_rows(None).await.unwrap(), 1_000);
}
fn guarded_description(dir: &tempfile::TempDir) -> crate::catalog::TableDescription {
crate::catalog::TableDescription {
location: dir.path().to_str().unwrap().to_string(),
storage_options: Default::default(),
managed_versioning: false,
}
}
/// The table's directory in the store's own paths.
fn table_root(dataset: &Dataset) -> Path {
let manifest = &dataset.manifest_location().path;
manifest
.parts()
.take(manifest.parts().count() - 2)
.collect()
}
/// A multipart upload started before the drop does not complete after it.
#[tokio::test(flavor = "multi_thread", worker_threads = 4)]
async fn a_multipart_upload_does_not_complete_into_a_dropped_table() {
let dir = tempfile::TempDir::new().unwrap();
table(&dir, 1).await;
let s3_like: Arc<dyn WrappingObjectStore> = Arc::new(S3Like::default());
let job = crate::dataset::open_guarded(&guarded_description(&dir), Some(s3_like))
.await
.unwrap();
let store = job.object_store(None).await.unwrap().inner.clone();
let path = table_root(&job).child("data").child("late.lance");
let mut upload = store.put_multipart(&path).await.unwrap();
upload
.put_part(PutPayload::from(vec![0u8; 16]))
.await
.unwrap();
std::fs::remove_dir_all(dir.path()).unwrap();
let completed = upload.complete().await;
assert!(
completed.is_err(),
"an upload completed into a dropped table"
);
assert!(files(dir.path()).is_empty(), "{:?}", files(dir.path()));
}
/// An upload whose last probe fails is aborted, not left with its parts.
#[tokio::test(flavor = "multi_thread", worker_threads = 4)]
async fn a_failed_probe_aborts_an_upload() {
let dir = tempfile::TempDir::new().unwrap();
table(&dir, 1).await;
let s3_like = S3Like::default();
let (armed, aborted) = (s3_like.armed.clone(), s3_like.aborted.clone());
let s3_like: Arc<dyn WrappingObjectStore> = Arc::new(s3_like);
let job = crate::dataset::open_guarded(&guarded_description(&dir), Some(s3_like))
.await
.unwrap();
let store = job.object_store(None).await.unwrap().inner.clone();
let path = table_root(&job).child("data").child("late.lance");
let mut upload = store.put_multipart(&path).await.unwrap();
upload
.put_part(PutPayload::from(vec![0u8; 16]))
.await
.unwrap();
armed.store(true, Ordering::SeqCst);
assert!(upload.complete().await.is_err());
assert!(aborted.load(Ordering::SeqCst), "the upload was left open");
}
/// A table created again, through the version a job just wrote, between
/// that write and its check keeps its own manifest at that path.
#[tokio::test(flavor = "multi_thread", worker_threads = 4)]
async fn a_rollback_keeps_the_manifest_of_a_table_created_again() {
let dir = tempfile::TempDir::new().unwrap();
table(&dir, 1).await;
let s3_like: Arc<dyn WrappingObjectStore> = Arc::new(S3Like {
recreate: Some((dir.path().to_path_buf(), Arc::new(AtomicBool::new(false)))),
..Default::default()
});
let mut job = crate::dataset::open_guarded(&guarded_description(&dir), Some(s3_like))
.await
.unwrap();
assert_eq!(job.version().version, 1);
assert!(
job.update_config([("seaweedfs.test", "job")])
.await
.is_err()
);
let again = Dataset::open(dir.path().to_str().unwrap()).await.unwrap();
assert_eq!(again.version().version, 2);
assert_eq!(
again.config().get("seaweedfs.test").map(String::as_str),
Some("again")
);
}
/// A table created again whose manifest at the job's path cannot be told
/// from the job's, because the HEAD fails, keeps it.
#[tokio::test(flavor = "multi_thread", worker_threads = 4)]
async fn a_failed_rollback_probe_keeps_the_manifest_of_a_table_created_again() {
let dir = tempfile::TempDir::new().unwrap();
table(&dir, 1).await;
let s3_like: Arc<dyn WrappingObjectStore> = Arc::new(S3Like {
recreate: Some((dir.path().to_path_buf(), Arc::new(AtomicBool::new(false)))),
fail_head_after_manifest: true,
heads_before_failing: 1,
..Default::default()
});
let mut job = crate::dataset::open_guarded(&guarded_description(&dir), Some(s3_like))
.await
.unwrap();
assert!(
job.update_config([("seaweedfs.test", "job")])
.await
.is_err()
);
let again = Dataset::open(dir.path().to_str().unwrap()).await.unwrap();
assert_eq!(again.version().version, 2);
assert_eq!(
again.config().get("seaweedfs.test").map(String::as_str),
Some("again")
);
}
/// A cleanup that picked its paths before the drop does not delete the files
/// of the table created again there.
#[tokio::test(flavor = "multi_thread", worker_threads = 4)]
async fn a_cleanup_does_not_delete_a_table_created_again() {
let dir = tempfile::TempDir::new().unwrap();
let mut dataset = table(&dir, 1).await;
dataset
.update_config([("seaweedfs.test", "1")])
.await
.unwrap();
let job = crate::dataset::open_guarded(&guarded_description(&dir), None)
.await
.unwrap();
let store = job.object_store(None).await.unwrap().inner.clone();
// The old version-1 manifest, as a cleanup would have picked it.
let mut first = dataset.checkout_version(1).await.unwrap();
let old = first.manifest_location().path.clone();
std::fs::remove_dir_all(dir.path()).unwrap();
let recreated = table(&dir, 1).await;
assert_eq!(recreated.manifest_location().path, old);
let deleted: Vec<_> = store
.delete_stream(futures::stream::iter(vec![Ok(old.clone())]).boxed())
.collect()
.await;
assert!(deleted.iter().any(|r| r.is_err()), "{deleted:?}");
first = Dataset::open(dir.path().to_str().unwrap()).await.unwrap();
assert_eq!(first.version().version, 1);
}
/// A probe that fails after a manifest was written says nothing about the
/// table: the manifest stays and the commit succeeds.
#[tokio::test(flavor = "multi_thread", worker_threads = 4)]
async fn a_failed_probe_does_not_take_back_a_commit() {
let dir = tempfile::TempDir::new().unwrap();
table(&dir, 1).await;
let flaky: Arc<dyn WrappingObjectStore> = Arc::new(S3Like {
fail_head_after_manifest: true,
..Default::default()
});
let mut job = crate::dataset::open_guarded(&guarded_description(&dir), Some(flaky))
.await
.unwrap();
let before = job.version().version;
job.update_config([("seaweedfs.test", "1")]).await.unwrap();
let reopened = Dataset::open(dir.path().to_str().unwrap()).await.unwrap();
assert_eq!(reopened.version().version, before + 1);
}
/// Behaves as an object store, not a file system: a multipart upload holds
/// its parts until it completes, and records an abort. Optionally fails one
/// HEAD after a manifest is written, as a timeout would, or creates the table
/// again, through two versions, right after the first manifest is written.
#[derive(Debug, Default)]
struct S3Like {
fail_head_after_manifest: bool,
/// HEADs let through after a manifest is written before one fails.
heads_before_failing: usize,
/// The next HEAD fails.
armed: Arc<AtomicBool>,
aborted: Arc<AtomicBool>,
recreate: Option<(PathBuf, Arc<AtomicBool>)>,
}
impl WrappingObjectStore for S3Like {
fn wrap(&self, _prefix: &str, original: Arc<dyn ObjectStore>) -> Arc<dyn ObjectStore> {
Arc::new(S3LikeStore {
inner: original,
fail_head_after_manifest: self.fail_head_after_manifest,
heads_before_failing: self.heads_before_failing,
heads_left: Arc::new(AtomicUsize::new(0)),
armed: self.armed.clone(),
aborted: self.aborted.clone(),
recreate: self.recreate.clone(),
})
}
}
#[derive(Debug)]
struct S3LikeStore {
inner: Arc<dyn ObjectStore>,
fail_head_after_manifest: bool,
heads_before_failing: usize,
heads_left: Arc<AtomicUsize>,
armed: Arc<AtomicBool>,
aborted: Arc<AtomicBool>,
recreate: Option<(PathBuf, Arc<AtomicBool>)>,
}
impl fmt::Display for S3LikeStore {
fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
write!(f, "S3Like({})", self.inner)
}
}
#[async_trait::async_trait]
impl ObjectStore for S3LikeStore {
async fn put_opts(
&self,
location: &Path,
payload: PutPayload,
opts: PutOptions,
) -> Result<PutResult> {
let put = self.inner.put_opts(location, payload, opts).await?;
if location.extension() == Some("manifest") {
if self.fail_head_after_manifest {
self.heads_left
.store(self.heads_before_failing, Ordering::SeqCst);
self.armed.store(true, Ordering::SeqCst);
}
if let Some((dir, fired)) = &self.recreate
&& !fired.swap(true, Ordering::SeqCst)
{
std::fs::remove_dir_all(dir).unwrap();
let mut again = create_empty(dir).await;
again
.update_config([("seaweedfs.test", "again")])
.await
.unwrap();
}
}
Ok(put)
}
async fn put_multipart_opts(
&self,
location: &Path,
_opts: PutMultipartOptions,
) -> Result<Box<dyn MultipartUpload>> {
Ok(Box::new(HeldParts {
inner: self.inner.clone(),
location: location.clone(),
parts: Vec::new(),
aborted: self.aborted.clone(),
}))
}
async fn get_opts(&self, location: &Path, options: GetOptions) -> Result<GetResult> {
if options.head
&& self.armed.load(Ordering::SeqCst)
&& self
.heads_left
.try_update(Ordering::SeqCst, Ordering::SeqCst, |n| n.checked_sub(1))
.is_err()
&& self.armed.swap(false, Ordering::SeqCst)
{
return Err(object_store::Error::Generic {
store: "S3Like",
source: "timed out".into(),
});
}
self.inner.get_opts(location, options).await
}
async fn get_ranges(&self, location: &Path, ranges: &[Range<u64>]) -> Result<Vec<Bytes>> {
self.inner.get_ranges(location, ranges).await
}
fn delete_stream(
&self,
locations: BoxStream<'static, Result<Path>>,
) -> BoxStream<'static, Result<Path>> {
self.inner.delete_stream(locations)
}
fn list(&self, prefix: Option<&Path>) -> BoxStream<'static, Result<ObjectMeta>> {
self.inner.list(prefix)
}
fn list_with_offset(
&self,
prefix: Option<&Path>,
offset: &Path,
) -> BoxStream<'static, Result<ObjectMeta>> {
self.inner.list_with_offset(prefix, offset)
}
async fn list_with_delimiter(&self, prefix: Option<&Path>) -> Result<ListResult> {
self.inner.list_with_delimiter(prefix).await
}
async fn copy_opts(&self, from: &Path, to: &Path, options: CopyOptions) -> Result<()> {
self.inner.copy_opts(from, to, options).await
}
async fn rename_opts(&self, from: &Path, to: &Path, options: RenameOptions) -> Result<()> {
self.inner.rename_opts(from, to, options).await
}
}
/// Parts held outside the table until the upload completes, as S3 does.
#[derive(Debug)]
struct HeldParts {
inner: Arc<dyn ObjectStore>,
location: Path,
parts: Vec<Bytes>,
aborted: Arc<AtomicBool>,
}
#[async_trait::async_trait]
impl MultipartUpload for HeldParts {
fn put_part(&mut self, data: PutPayload) -> object_store::UploadPart {
self.parts.extend(data);
Box::pin(async { Ok(()) })
}
async fn complete(&mut self) -> Result<PutResult> {
let payload: PutPayload = self.parts.drain(..).collect();
self.inner.put(&self.location, payload).await
}
async fn abort(&mut self) -> Result<()> {
self.parts.clear();
self.aborted.store(true, Ordering::SeqCst);
Ok(())
}
}
/// An empty table at `dir`.
async fn create_empty(dir: &std::path::Path) -> Dataset {
Dataset::write(
arrow_array::RecordBatchIterator::new(
Vec::<std::result::Result<arrow_array::RecordBatch, arrow_schema::ArrowError>>::new(),
Arc::new(crate::metadata_index::tests::lifecycle::base_schema()),
),
dir.to_str().unwrap(),
None,
)
.await
.unwrap()
}
+163 -30
View File
@@ -3,11 +3,12 @@ use std::collections::HashMap;
use anyhow::{Context, Result, anyhow};
use async_trait::async_trait;
use chrono::{Duration, Utc};
use lance::Dataset;
use lance::dataset::cleanup::{CleanupPolicy, cleanup_old_versions};
use seaweed_worker_core::config_form::{form, int_or, int_value, number_field};
use seaweed_worker_core::pb::{
ConfigValue, DetectionComplete, DetectionProposals, ExecuteJobRequest, JobCompleted,
JobProgressUpdate, JobProposal, JobResult, JobTypeCapability, JobTypeDescriptor,
AdminRuntimeDefaults, ConfigValue, DetectionComplete, DetectionProposals, ExecuteJobRequest,
JobCompleted, JobProgressUpdate, JobProposal, JobResult, JobTypeCapability, JobTypeDescriptor,
RunDetectionRequest,
};
use seaweed_worker_core::{DetectionSender, ExecutionSender, JobHandler};
@@ -15,6 +16,7 @@ use tracing::warn;
use crate::catalog::{NamespaceClient, SigningKey, parse_id};
use crate::dataset;
use crate::jobs::purge::{self, Retention};
use crate::jobs::{clamp, string_list, table_id};
pub const JOB_TYPE: &str = "lance_cleanup_versions";
@@ -28,7 +30,9 @@ const MAX_MIN_VERSIONS: i64 = 1000;
/// Lance keeps every version until something removes it. Lance can also do this
/// itself through auto-cleanup, so this job is for deployments that would rather
/// the cluster owned the policy than each writer.
/// the cluster owned the policy than each writer. A table may override the
/// retention in its own config, and may ask for a purge (see [`purge`]), which
/// this job carries out ahead of the retention window.
pub struct CleanupVersionsHandler {
namespace_url: String,
storage: dataset::Storage,
@@ -68,9 +72,8 @@ impl CleanupVersionsHandler {
/// The oldest version that may be removed while still leaving `min_versions`
/// behind, or None when the table has no more than the floor.
async fn version_floor(table: &dataset::OpenTable, min_versions: usize) -> Result<Option<u64>> {
let mut versions: Vec<u64> = table
.dataset
async fn version_floor(dataset: &Dataset, min_versions: usize) -> Result<Option<u64>> {
let mut versions: Vec<u64> = dataset
.versions()
.await?
.iter()
@@ -83,6 +86,33 @@ async fn version_floor(table: &dataset::OpenTable, min_versions: usize) -> Resul
Ok(Some(versions[versions.len() - min_versions]))
}
/// The cleanup that keeps `min_versions` and every version younger than
/// `retain_hours`, or None when the floor leaves nothing to remove. A purge
/// cleans up through it too.
pub async fn retention_policy(
dataset: &Dataset,
retain_hours: i64,
min_versions: usize,
) -> Result<Option<CleanupPolicy>> {
let Some(floor) = version_floor(dataset, min_versions).await? else {
return Ok(None);
};
Ok(Some(CleanupPolicy {
before_timestamp: Some(Utc::now() - Duration::hours(retain_hours)),
// should_clean() ANDs its clauses, so a version has to be both older
// than the window and below the floor to go.
before_version: Some(floor),
// Files this dataset cannot account for are left alone: they may
// belong to a writer that has not committed yet, and deleting them
// would corrupt a commit in flight.
delete_unverified: false,
// A tagged version is pinned on purpose, so refuse rather than
// silently dropping what someone named.
error_if_tagged_old_versions: true,
..Default::default()
}))
}
#[async_trait]
impl JobHandler for CleanupVersionsHandler {
fn capability(&self) -> JobTypeCapability {
@@ -134,6 +164,12 @@ impl JobHandler for CleanupVersionsHandler {
defaults.clone(),
)),
worker_default_values: defaults,
// A purge compacts and retrains, so it gets a build's time.
admin_runtime_defaults: Some(AdminRuntimeDefaults {
job_type_max_runtime_seconds: super::vector_index::EXECUTION_TIMEOUT_SECONDS,
execution_timeout_seconds: super::vector_index::EXECUTION_TIMEOUT_SECONDS,
..Default::default()
}),
..Default::default()
}
}
@@ -181,10 +217,23 @@ impl JobHandler for CleanupVersionsHandler {
continue;
}
};
let config = table.dataset.config();
let pending_purge = purge::pending(config);
let floor = Retention::for_table(
config,
Retention {
retain_hours: DEFAULT_RETAIN_HOURS,
min_versions,
},
MAX_RETAIN_HOURS,
MAX_MIN_VERSIONS,
)
.min_versions;
// Age is decided at execution against the retention window; a table
// at or under the floor cannot lose a version whatever its age, so
// proposing one would only produce a job with nothing to do.
if stats.total_versions <= min_versions {
// proposing one would only produce a job with nothing to do. A
// requested purge ignores the floor: it keeps only the latest.
if pending_purge.is_none() && stats.total_versions <= floor {
continue;
}
let mut parameters: HashMap<String, ConfigValue> = HashMap::new();
@@ -193,12 +242,21 @@ impl JobHandler for CleanupVersionsHandler {
proposal_id: format!("{JOB_TYPE}:{encoded}"),
dedupe_key: format!("{JOB_TYPE}:{encoded}"),
job_type: JOB_TYPE.to_string(),
summary: format!("Clean up {encoded} ({} versions)", stats.total_versions),
detail: format!(
"{} versions retained, above the {min_versions} floor; \
those outside the retention window can go",
stats.total_versions
),
summary: match &pending_purge {
Some(_) => format!("Purge {encoded} ({} versions)", stats.total_versions),
None => format!("Clean up {encoded} ({} versions)", stats.total_versions),
},
detail: match &pending_purge {
Some(token) => format!(
"purge requested at {token}: rewrite fragments with deletions \
and keep only the latest version"
),
None => format!(
"{} versions retained, above the {floor} floor; \
those outside the retention window can go",
stats.total_versions
),
},
parameters,
..Default::default()
});
@@ -254,14 +312,33 @@ impl JobHandler for CleanupVersionsHandler {
) as usize;
let client = self.client();
let table = dataset::open(&client, &id, &self.storage).await?;
let mut table = dataset::open(&client, &id, &self.storage).await?;
let before = table.stats().await?;
if let Some(token) = purge::pending(table.dataset.config()) {
return self
.execute_purge(request, sender, &job.job_id, &mut table, &token)
.await;
}
let Retention {
retain_hours,
min_versions,
} = Retention::for_table(
table.dataset.config(),
Retention {
retain_hours,
min_versions,
},
MAX_RETAIN_HOURS,
MAX_MIN_VERSIONS,
);
// The floor is a promise about how much history survives, so it has to
// be applied here and not only when the job was proposed: by the time it
// runs, versions may have aged past the retention window, and age alone
// would take the table below what the operator asked to keep.
let Some(floor) = version_floor(&table, min_versions).await? else {
let Some(policy) = retention_policy(&table.dataset, retain_hours, min_versions).await?
else {
sender.send_completed(JobCompleted {
request_id: request.request_id.clone(),
job_id: job.job_id.clone(),
@@ -288,20 +365,6 @@ impl JobHandler for CleanupVersionsHandler {
..Default::default()
})?;
let policy = CleanupPolicy {
before_timestamp: Some(Utc::now() - Duration::hours(retain_hours)),
// should_clean() ANDs its clauses, so a version has to be both older
// than the window and below the floor to go.
before_version: Some(floor),
// Files this dataset cannot account for are left alone: they may
// belong to a writer that has not committed yet, and deleting them
// would corrupt a commit in flight.
delete_unverified: false,
// A tagged version is pinned on purpose, so refuse rather than
// silently dropping what someone named.
error_if_tagged_old_versions: true,
..Default::default()
};
let stats = cleanup_old_versions(&table.dataset, policy)
.await
.with_context(|| format!("clean up versions of {}", table.location))?;
@@ -343,3 +406,73 @@ impl JobHandler for CleanupVersionsHandler {
Ok(())
}
}
impl CleanupVersionsHandler {
async fn execute_purge(
&self,
request: &ExecuteJobRequest,
sender: &dyn ExecutionSender,
job_id: &str,
table: &mut dataset::OpenTable,
token: &str,
) -> Result<()> {
let stats = purge::purge(&mut table.dataset, token)
.await
.with_context(|| format!("purge {}", table.location))?;
if let Some(counters) = &self.metrics {
counters.versions_removed.inc_by(stats.old_versions);
counters.bytes_reclaimed.inc_by(stats.bytes_removed);
}
let mut output: HashMap<String, ConfigValue> = HashMap::new();
output.insert(
"fragments_rewritten".to_string(),
int_value(stats.fragments_rewritten as i64),
);
output.insert(
"old_versions_removed".to_string(),
int_value(stats.old_versions as i64),
);
output.insert(
"bytes_removed".to_string(),
int_value(stats.bytes_removed as i64),
);
sender.send_completed(JobCompleted {
request_id: request.request_id.clone(),
job_id: job_id.to_string(),
job_type: JOB_TYPE.to_string(),
success: true,
result: Some(JobResult {
output_values: output,
summary: if stats.served {
format!(
"purged: rewrote {} fragments, removed {} versions and {} bytes",
stats.fragments_rewritten, stats.old_versions, stats.bytes_removed
)
} else {
format!(
"purge pending: rewrote {} fragments, removed {} versions and {} bytes; \
versions from before the rebuild are still retained",
stats.fragments_rewritten, stats.old_versions, stats.bytes_removed
)
},
..Default::default()
}),
..Default::default()
})?;
Ok(())
}
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn a_purge_gets_a_builds_execution_timeout() {
let descriptor = CleanupVersionsHandler::new(String::new()).descriptor();
let defaults = descriptor.admin_runtime_defaults.unwrap();
assert_eq!(defaults.execution_timeout_seconds, 3600);
assert_eq!(defaults.job_type_max_runtime_seconds, 3600);
assert!(!defaults.enabled);
}
}
@@ -0,0 +1,757 @@
use std::collections::HashMap;
use anyhow::{Result, anyhow, bail};
use async_trait::async_trait;
use seaweed_worker_core::config_form::{form, int_or, int_value, number_field, string_value};
use seaweed_worker_core::pb::{
AdminRuntimeDefaults, ConfigValue, DetectionComplete, DetectionProposals, ExecuteJobRequest,
JobCompleted, JobProgressUpdate, JobProposal, JobResult, JobTypeCapability, JobTypeDescriptor,
RunDetectionRequest, config_value::Kind,
};
use seaweed_worker_core::{DetectionSender, ExecutionSender, JobHandler};
use tracing::warn;
use crate::catalog::{NamespaceClient, SigningKey, parse_id};
use crate::dataset;
use crate::jobs::vector_index::{
EXECUTION_TIMEOUT_SECONDS, declare_build_runtime, is_vector_table,
};
use crate::jobs::{clamp, string_list, table_id};
use crate::metadata_index::{self, DEFAULT_MIN_UNINDEXED_ROWS, DEFAULT_SEGMENT_ROWS, State};
pub const JOB_TYPE: &str = "lance_metadata_index";
const SEGMENT_ROWS_FLOOR: i64 = 1_000;
const SEGMENT_ROWS_CEILING: i64 = 1_000_000_000;
const MIN_UNINDEXED_FLOOR: i64 = 0;
const MIN_UNINDEXED_CEILING: i64 = 100_000_000;
/// What a job does to one key.
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub enum Action {
/// Add the requested key's column, or finish a `building` one.
Add,
/// Drop an `off` key's index and column.
Remove,
/// Fill NULL rows and fold unindexed fragments of a `ready` key.
Maintain,
}
impl Action {
pub fn as_str(self) -> &'static str {
match self {
Action::Add => "add",
Action::Remove => "remove",
Action::Maintain => "maintain",
}
}
pub fn parse(s: &str) -> Option<Self> {
match s {
"add" => Some(Action::Add),
"remove" => Some(Action::Remove),
"maintain" => Some(Action::Maintain),
_ => None,
}
}
}
/// The work an index's table asks for: requested and `building` keys to add,
/// `off` keys to remove, and `ready` keys whose unindexed rows reached
/// `min_unindexed_rows`.
pub async fn pending(
dataset: &lance::Dataset,
min_unindexed_rows: u64,
) -> Result<Vec<(String, Action)>> {
let mut out: Vec<(String, Action)> = metadata_index::requests(dataset)
.into_iter()
.map(|key| (key, Action::Add))
.collect();
for key in metadata_index::keys_of(dataset) {
let action = match key.state {
Some(State::Building) => Action::Add,
Some(State::Off) => Action::Remove,
Some(State::Ready) => {
let segments = metadata_index::segments(dataset, &key.column).await?;
let pending = metadata_index::unindexed(dataset, &segments);
let unindexed: u64 = pending.iter().map(|(_, rows)| rows).sum();
if unindexed == 0 {
continue;
}
// Rows a write left NULL (one that raced the key's add and
// stopped before fixing them) wait for no threshold: queries
// through the key miss them until they are derived again.
// Nor does the first segment of a key that went ready on an
// empty table.
if unindexed < min_unindexed_rows
&& !segments.is_empty()
&& !has_nulls(dataset, &key.column, &pending).await?
{
continue;
}
Action::Maintain
}
None => continue,
};
out.push((key.key, action));
}
Ok(out)
}
/// Whether `fragments` hold a row whose `column` is NULL.
async fn has_nulls(
dataset: &lance::Dataset,
column: &str,
fragments: &[(u32, u64)],
) -> Result<bool> {
let ids: Vec<u64> = fragments.iter().map(|(id, _)| *id as u64).collect();
let mut scanner = dataset.scan();
scanner
.filter(&format!("`{}` IS NULL", column.replace('`', "``")))?
.with_fragments(
dataset
.fragments()
.iter()
.filter(|f| ids.contains(&f.id))
.cloned()
.collect(),
);
Ok(scanner.count_rows().await? > 0)
}
/// Carries out `action` on `key`, as the table now records it; an action the
/// table no longer asks for is skipped. Returns what was done.
pub async fn run(
dataset: &mut lance::Dataset,
key: &str,
action: Action,
segment_rows: u64,
min_unindexed_rows: u64,
grace: std::time::Duration,
) -> Result<String> {
let column = metadata_index::keys_of(dataset)
.into_iter()
.find(|k| k.key == key);
match (action, column) {
(Action::Add, None) => {
if !metadata_index::requests(dataset).iter().any(|k| k == key) {
return Ok(format!("{key} is no longer requested"));
}
let added = metadata_index::add_column(dataset, key).await?;
if added.state != Some(State::Building) {
return Ok(format!(
"{key} was withdrawn while {} was added",
added.column
));
}
let filled = metadata_index::finish_after(dataset, &added, segment_rows, grace).await?;
Ok(format!(
"{key} indexed in {} ({filled} rows a racing write left derived again)",
added.column
))
}
(Action::Add, Some(column)) if column.state == Some(State::Building) => {
// An add that stopped before setting again what its commit
// overwrote leaves that to the build resuming it.
let state = metadata_index::recover_add(dataset, &column).await?;
if state != Some(State::Building) {
return Ok(format!(
"{key} is {} after its add was recovered",
state.map_or("unknown", State::as_str)
));
}
let filled =
metadata_index::finish_after(dataset, &column, segment_rows, grace).await?;
Ok(format!(
"{key} indexed in {} ({filled} rows a racing write left derived again)",
column.column
))
}
(Action::Remove, Some(column)) if column.state == Some(State::Off) => {
// Dropping the column drops its restore mark: set again first
// what its add overwrote.
metadata_index::recover_add(dataset, &column).await?;
metadata_index::remove_column(dataset, &column.column).await?;
Ok(format!("{key} removed with {}", column.column))
}
(Action::Maintain, Some(column)) if column.state == Some(State::Ready) => {
let (filled, folded) =
metadata_index::maintain(dataset, &column, segment_rows, min_unindexed_rows)
.await?;
Ok(format!("{key}: {folded:?}, {filled} rows derived again"))
}
(action, column) => Ok(format!(
"{key} needs no {}: {}",
action.as_str(),
column
.and_then(|c| c.state)
.map_or("not indexed", State::as_str)
)),
}
}
/// Adds, maintains and removes the indexed metadata keys of S3 Vectors
/// indexes, as each index's table records them. The admin dispatches a job
/// when an operator asks; detection finds the rest.
pub struct MetadataIndexHandler {
namespace_url: String,
storage: dataset::Storage,
signing_key: Option<SigningKey>,
metrics: Option<crate::metrics::LanceMetrics>,
}
impl MetadataIndexHandler {
pub fn new(namespace_url: String) -> Self {
Self {
namespace_url,
storage: dataset::Storage::default(),
signing_key: None,
metrics: None,
}
}
pub fn with_metrics(mut self, metrics: Option<crate::metrics::LanceMetrics>) -> Self {
self.metrics = metrics;
self
}
pub fn with_storage(mut self, storage: dataset::Storage) -> Self {
self.storage = storage;
self
}
pub fn with_signing_key(mut self, signing_key: Option<SigningKey>) -> Self {
self.signing_key = signing_key;
self
}
fn client(&self) -> NamespaceClient {
NamespaceClient::new(self.namespace_url.clone()).with_signing_key(self.signing_key.as_ref())
}
}
fn settings(values: &HashMap<String, ConfigValue>) -> (u64, u64) {
(
clamp(
int_or(values, "segment_rows", DEFAULT_SEGMENT_ROWS as i64),
SEGMENT_ROWS_FLOOR,
SEGMENT_ROWS_CEILING,
) as u64,
clamp(
int_or(
values,
"min_unindexed_rows",
DEFAULT_MIN_UNINDEXED_ROWS as i64,
),
MIN_UNINDEXED_FLOOR,
MIN_UNINDEXED_CEILING,
) as u64,
)
}
fn string_param(parameters: &HashMap<String, ConfigValue>, name: &str) -> Option<String> {
match parameters.get(name)?.kind.as_ref()? {
Kind::StringValue(s) => Some(s.clone()),
_ => None,
}
}
#[async_trait]
impl JobHandler for MetadataIndexHandler {
fn capability(&self) -> JobTypeCapability {
JobTypeCapability {
job_type: JOB_TYPE.to_string(),
can_detect: true,
can_execute: true,
max_detection_concurrency: 1,
max_execution_concurrency: 1,
display_name: "Lance Metadata Index".to_string(),
description: "Add, maintain and remove the indexed metadata keys of S3 Vectors indexes"
.to_string(),
weight: 40,
}
}
fn descriptor(&self) -> JobTypeDescriptor {
let mut defaults: HashMap<String, ConfigValue> = HashMap::new();
defaults.insert(
"segment_rows".to_string(),
int_value(DEFAULT_SEGMENT_ROWS as i64),
);
defaults.insert(
"min_unindexed_rows".to_string(),
int_value(DEFAULT_MIN_UNINDEXED_ROWS as i64),
);
JobTypeDescriptor {
job_type: JOB_TYPE.to_string(),
display_name: "Lance Metadata Index".to_string(),
description:
"Keep a LABEL_LIST index for each metadata key an S3 Vectors index asks for"
.to_string(),
icon: "fas fa-tags".to_string(),
descriptor_version: 1,
worker_config_form: Some(form(
"lance-metadata-index-worker",
"Metadata index",
vec![
number_field(
"segment_rows",
"Segment rows",
"Rows an index segment takes before a new one starts",
SEGMENT_ROWS_FLOOR,
SEGMENT_ROWS_CEILING,
),
number_field(
"min_unindexed_rows",
"Unindexed rows",
"Rows written since the last fold that wait for the next one",
MIN_UNINDEXED_FLOOR,
MIN_UNINDEXED_CEILING,
),
],
defaults.clone(),
)),
worker_default_values: defaults,
admin_runtime_defaults: Some(AdminRuntimeDefaults {
job_type_max_runtime_seconds: EXECUTION_TIMEOUT_SECONDS,
execution_timeout_seconds: EXECUTION_TIMEOUT_SECONDS,
..Default::default()
}),
..Default::default()
}
}
async fn detect(
&self,
request: &RunDetectionRequest,
sender: &dyn DetectionSender,
) -> Result<()> {
let (_, min_unindexed_rows) = settings(&request.worker_config_values);
let max_results = usize::try_from(request.max_results).unwrap_or(0);
let client = self.client();
let mut proposals = Vec::new();
let mut has_more = false;
'tables: for encoded in &client.list_all_tables().await? {
let id = parse_id(encoded);
if !is_vector_table(&id) {
continue;
}
let table = match dataset::open(&client, &id, &self.storage).await {
Ok(table) => table,
Err(err) => {
if let Some(counters) = &self.metrics {
counters.worker.object_skipped(JOB_TYPE, "open");
}
warn!("skipping {encoded}: {err:#}");
continue;
}
};
if let Some(counters) = &self.metrics {
counters.worker.object_seen(JOB_TYPE);
}
let work = match pending(&table.dataset, min_unindexed_rows).await {
Ok(work) => work,
Err(err) => {
warn!("skipping {encoded}: reading its indexed keys failed: {err:#}");
continue;
}
};
for (key, action) in work {
if max_results > 0 && proposals.len() >= max_results {
has_more = true;
break 'tables;
}
proposals.push(proposal(&id, encoded, &key, action));
}
}
let total = proposals.len() as i32;
sender.send_proposals(DetectionProposals {
request_id: request.request_id.clone(),
job_type: JOB_TYPE.to_string(),
proposals,
has_more,
})?;
sender.send_complete(DetectionComplete {
request_id: request.request_id.clone(),
job_type: JOB_TYPE.to_string(),
success: true,
error_message: String::new(),
total_proposals: total,
skipped_reason: String::new(),
eligible_after: None,
})?;
Ok(())
}
async fn execute(
&self,
request: &ExecuteJobRequest,
sender: &dyn ExecutionSender,
) -> Result<()> {
let job = request
.job
.as_ref()
.ok_or_else(|| anyhow!("execute request carried no job"))?;
let id = table_id(&job.parameters)
.ok_or_else(|| anyhow!("job {} carried no table_id", job.job_id))?;
if !is_vector_table(&id) {
bail!("{} is not an S3 Vectors index", id.join("/"));
}
let key = string_param(&job.parameters, "key")
.ok_or_else(|| anyhow!("job {} carried no key", job.job_id))?;
let action = string_param(&job.parameters, "action")
.as_deref()
.and_then(Action::parse)
.ok_or_else(|| anyhow!("job {} carried no valid action", job.job_id))?;
let (segment_rows, min_unindexed_rows) = settings(&request.worker_config_values);
sender.send_progress(JobProgressUpdate {
request_id: request.request_id.clone(),
job_id: job.job_id.clone(),
job_type: JOB_TYPE.to_string(),
progress_percent: 10.0,
stage: format!("{} {key}", action.as_str()),
..Default::default()
})?;
let client = self.client();
let mut table = dataset::open(&client, &id, &self.storage).await?;
let summary = run(
&mut table.dataset,
&key,
action,
segment_rows,
min_unindexed_rows,
metadata_index::RACE_GRACE,
)
.await?;
sender.send_completed(JobCompleted {
request_id: request.request_id.clone(),
job_id: job.job_id.clone(),
job_type: JOB_TYPE.to_string(),
success: true,
result: Some(JobResult {
summary,
..Default::default()
}),
..Default::default()
})?;
Ok(())
}
}
/// The proposal for one key of one table; the admin builds the same
/// parameters when it dispatches a job for an operator.
fn proposal(id: &[String], encoded: &str, key: &str, action: Action) -> JobProposal {
let mut parameters: HashMap<String, ConfigValue> = HashMap::new();
parameters.insert("table_id".to_string(), string_list(id));
parameters.insert("key".to_string(), string_value(key));
parameters.insert("action".to_string(), string_value(action.as_str()));
declare_build_runtime(&mut parameters);
let dedupe = format!("{JOB_TYPE}:{encoded}:{key}:{}", action.as_str());
JobProposal {
proposal_id: dedupe.clone(),
dedupe_key: dedupe,
job_type: JOB_TYPE.to_string(),
summary: format!("{} metadata key {key} of {encoded}", action.as_str()),
parameters,
..Default::default()
}
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn a_proposal_names_the_table_key_and_action() {
let id = parse_id("media$vector$movies");
let p = proposal(&id, "media$vector$movies", "author", Action::Remove);
assert_eq!(table_id(&p.parameters), Some(id));
assert_eq!(
string_param(&p.parameters, "key").as_deref(),
Some("author")
);
assert_eq!(
string_param(&p.parameters, "action")
.as_deref()
.and_then(Action::parse),
Some(Action::Remove)
);
assert_eq!(
p.dedupe_key,
"lance_metadata_index:media$vector$movies:author:remove"
);
}
#[test]
fn settings_are_held_to_their_ranges() {
let mut values = HashMap::new();
assert_eq!(
settings(&values),
(DEFAULT_SEGMENT_ROWS, DEFAULT_MIN_UNINDEXED_ROWS)
);
values.insert("segment_rows".to_string(), int_value(-5));
values.insert("min_unindexed_rows".to_string(), int_value(-1));
assert_eq!(settings(&values), (SEGMENT_ROWS_FLOOR as u64, 0));
}
/// A removal of a key whose add stopped before setting again what it
/// overwrote sets that first: a key requested meanwhile stays requested.
#[tokio::test(flavor = "multi_thread", worker_threads = 4)]
async fn a_removal_keeps_what_the_add_overwrote() {
use crate::metadata_index::tests::lifecycle::table;
let dir = tempfile::TempDir::new().unwrap();
let mut dataset = table(&dir, 2).await;
metadata_index::request(&mut dataset, "author")
.await
.unwrap();
let mut adder = dataset.clone();
metadata_index::request(&mut dataset, "lang").await.unwrap();
metadata_index::commit_column(&mut adder, "author")
.await
.unwrap();
metadata_index::request_removal(&mut adder, "author")
.await
.unwrap();
run(
&mut adder,
"author",
Action::Remove,
1_000,
0,
std::time::Duration::ZERO,
)
.await
.unwrap();
assert!(metadata_index::keys_of(&adder).is_empty());
assert_eq!(metadata_index::requests(&adder), ["lang"]);
}
/// A withdrawal of a request an interrupted add overwrote sets that
/// first, so recovering the add does not bring the request back.
#[tokio::test(flavor = "multi_thread", worker_threads = 4)]
async fn a_withdrawal_of_a_request_an_add_overwrote_stays_withdrawn() {
use crate::metadata_index::tests::lifecycle::table;
let dir = tempfile::TempDir::new().unwrap();
let mut dataset = table(&dir, 2).await;
metadata_index::request(&mut dataset, "author")
.await
.unwrap();
let mut adder = dataset.clone();
metadata_index::request(&mut dataset, "lang").await.unwrap();
metadata_index::commit_column(&mut adder, "author")
.await
.unwrap();
dataset.checkout_latest().await.unwrap();
metadata_index::request_removal(&mut dataset, "lang")
.await
.unwrap();
let author = metadata_index::keys_of(&dataset).remove(0);
metadata_index::recover_add(&mut dataset, &author)
.await
.unwrap();
assert!(metadata_index::requests(&dataset).is_empty());
}
/// A withdrawal that read the request before the key's column was added
/// and committed after sets the column off.
#[tokio::test(flavor = "multi_thread", worker_threads = 4)]
async fn a_withdrawal_racing_the_add_sets_the_column_off() {
use crate::metadata_index::tests::lifecycle::table;
let dir = tempfile::TempDir::new().unwrap();
let mut dataset = table(&dir, 2).await;
metadata_index::request(&mut dataset, "genre")
.await
.unwrap();
let mut operator = dataset.clone();
let added = metadata_index::add_column(&mut dataset, "genre")
.await
.unwrap();
assert_eq!(added.state, Some(State::Building));
metadata_index::request_removal(&mut operator, "genre")
.await
.unwrap();
dataset.checkout_latest().await.unwrap();
assert_eq!(metadata_index::keys_of(&dataset)[0].state, Some(State::Off));
assert!(metadata_index::requests(&dataset).is_empty());
}
/// A key that went ready on an empty table builds its first segment
/// once rows arrive, below the threshold.
#[tokio::test(flavor = "multi_thread", worker_threads = 4)]
async fn a_key_ready_without_a_segment_is_maintained_at_once() {
use crate::metadata_index::tests::lifecycle::{table, upsert};
let dir = tempfile::TempDir::new().unwrap();
let mut dataset = table(&dir, 1).await;
dataset.delete("true").await.unwrap();
metadata_index::request(&mut dataset, "author")
.await
.unwrap();
run(
&mut dataset,
"author",
Action::Add,
1_000,
0,
std::time::Duration::ZERO,
)
.await
.unwrap();
assert_eq!(
metadata_index::keys_of(&dataset)[0].state,
Some(State::Ready)
);
assert!(pending(&dataset, 1_000_000).await.unwrap().is_empty());
let mut dataset = upsert(dataset, 0..100).await;
assert_eq!(
pending(&dataset, 1_000_000).await.unwrap(),
[("author".to_string(), Action::Maintain)]
);
run(
&mut dataset,
"author",
Action::Maintain,
1_000,
1_000_000,
std::time::Duration::ZERO,
)
.await
.unwrap();
assert_eq!(
metadata_index::segments(&dataset, "_mk_0")
.await
.unwrap()
.len(),
1
);
assert!(pending(&dataset, 1_000_000).await.unwrap().is_empty());
}
/// What the table asks for is what a job does, and a job the table no
/// longer asks for does nothing.
#[tokio::test(flavor = "multi_thread", worker_threads = 4)]
async fn jobs_follow_what_the_table_records() {
use crate::metadata_index::tests::lifecycle::{table, upsert};
let dir = tempfile::TempDir::new().unwrap();
let mut dataset = table(&dir, 4).await;
assert!(pending(&dataset, 0).await.unwrap().is_empty());
metadata_index::request(&mut dataset, "author")
.await
.unwrap();
metadata_index::request(&mut dataset, "genre")
.await
.unwrap();
assert_eq!(
pending(&dataset, 0).await.unwrap(),
[
("author".to_string(), Action::Add),
("genre".to_string(), Action::Add)
]
);
metadata_index::request_removal(&mut dataset, "genre")
.await
.unwrap();
let skipped = run(
&mut dataset,
"genre",
Action::Add,
1_000,
0,
std::time::Duration::ZERO,
)
.await
.unwrap();
assert_eq!(skipped, "genre is no longer requested");
run(
&mut dataset,
"author",
Action::Add,
1_000,
0,
std::time::Duration::ZERO,
)
.await
.unwrap();
let keys = metadata_index::keys_of(&dataset);
assert_eq!(keys.len(), 1);
assert_eq!(keys[0].state, Some(State::Ready));
assert_eq!(
metadata_index::segments(&dataset, "_mk_0")
.await
.unwrap()
.len(),
4
);
assert!(pending(&dataset, 0).await.unwrap().is_empty());
let mut dataset = upsert(dataset, 4_000..4_500).await;
assert!(pending(&dataset, 1_000).await.unwrap().is_empty());
assert_eq!(
pending(&dataset, 500).await.unwrap(),
[("author".to_string(), Action::Maintain)]
);
run(
&mut dataset,
"author",
Action::Maintain,
1_000,
500,
std::time::Duration::ZERO,
)
.await
.unwrap();
assert!(pending(&dataset, 0).await.unwrap().is_empty());
// A raced write that left NULL rows is maintained at once.
let stale = dataset.clone();
let mut dataset =
crate::metadata_index::tests::lifecycle::upsert_with_null_columns(stale, 4_500..4_520)
.await;
assert_eq!(
pending(&dataset, 1_000_000).await.unwrap(),
[("author".to_string(), Action::Maintain)]
);
run(
&mut dataset,
"author",
Action::Maintain,
1_000,
1_000_000,
std::time::Duration::ZERO,
)
.await
.unwrap();
assert!(pending(&dataset, 1_000_000).await.unwrap().is_empty());
let noop = run(
&mut dataset,
"author",
Action::Remove,
1_000,
0,
std::time::Duration::ZERO,
)
.await
.unwrap();
assert_eq!(noop, "author needs no remove: ready");
metadata_index::request_removal(&mut dataset, "author")
.await
.unwrap();
assert_eq!(
pending(&dataset, 0).await.unwrap(),
[("author".to_string(), Action::Remove)]
);
run(
&mut dataset,
"author",
Action::Remove,
1_000,
0,
std::time::Duration::ZERO,
)
.await
.unwrap();
assert!(metadata_index::keys_of(&dataset).is_empty());
assert!(pending(&dataset, 0).await.unwrap().is_empty());
}
}
@@ -9,6 +9,8 @@
pub mod cleanup;
pub mod compact;
pub mod indices;
pub mod metadata_index;
pub mod purge;
pub mod sort;
pub mod vector_index;
@@ -65,6 +67,12 @@ pub fn handlers(
.with_signing_key(signing_key.clone())
.with_metrics(metrics.clone()),
),
Arc::new(
metadata_index::MetadataIndexHandler::new(namespace_url.clone())
.with_storage(storage.clone())
.with_signing_key(signing_key.clone())
.with_metrics(metrics.clone()),
),
Arc::new(
sort::SortHandler::new(namespace_url.clone())
.with_storage(storage.clone())
@@ -0,0 +1,328 @@
//! Per-table retention for version cleanup, and purges: compaction that
//! rewrites every fragment with deletions followed by a cleanup that keeps
//! only the latest version, so the bytes of deleted rows leave storage now
//! rather than after the retention window.
//!
//! Both are set in the table's own Lance config, so any client that can write
//! the table can set them and they travel with it:
//!
//! - `seaweedfs.cleanup.retain_hours` and `seaweedfs.cleanup.min_versions`
//! override the worker's retention for this table.
//! - `seaweedfs.purge.requested` holds a request token ([`request_purge`]
//! writes the current time); a purge is pending while it differs from
//! `seaweedfs.purge.completed`, which the purge sets to the token it served.
//! `seaweedfs.purge.rebuilt` records the token whose rebuild is done.
use std::collections::{BTreeMap, BTreeSet, HashMap};
use anyhow::{Context, Result, bail};
use chrono::Utc;
use lance::Dataset;
use lance::dataset::cleanup::{RemovalStats, cleanup_old_versions};
use lance::dataset::optimize::{CompactionOptions, compact_files};
use lance::index::{DatasetIndexExt, DatasetIndexInternalExt};
use lance_index::metrics::NoOpMetricsCollector;
use lance_index::optimize::OptimizeOptions;
use lance_index::{IndexType, is_system_index};
use lance_table::format::IndexMetadata;
use tracing::warn;
use crate::jobs::cleanup::retention_policy;
pub const RETAIN_HOURS_KEY: &str = "seaweedfs.cleanup.retain_hours";
pub const MIN_VERSIONS_KEY: &str = "seaweedfs.cleanup.min_versions";
pub const PURGE_REQUESTED_KEY: &str = "seaweedfs.purge.requested";
pub const PURGE_COMPLETED_KEY: &str = "seaweedfs.purge.completed";
pub const PURGE_REBUILT_KEY: &str = "seaweedfs.purge.rebuilt";
/// The retention cleanup applies to one table: the worker's, unless the
/// table's config overrides a value with one inside the worker's range.
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub struct Retention {
pub retain_hours: i64,
pub min_versions: usize,
}
impl Retention {
pub fn for_table(
config: &HashMap<String, String>,
worker: Retention,
max_retain_hours: i64,
max_min_versions: i64,
) -> Retention {
let read = |key: &str, min: i64, max: i64| -> Option<i64> {
let raw = config.get(key)?;
match raw.trim().parse::<i64>() {
Ok(value) if (min..=max).contains(&value) => Some(value),
_ => {
warn!("ignoring table config {key}={raw:?}: not an integer in {min}..={max}");
None
}
}
};
Retention {
retain_hours: read(RETAIN_HOURS_KEY, 0, max_retain_hours)
.unwrap_or(worker.retain_hours),
min_versions: read(MIN_VERSIONS_KEY, 1, max_min_versions)
.map(|v| v as usize)
.unwrap_or(worker.min_versions),
}
}
}
/// The request token of a purge the table is waiting for, if any.
pub fn pending(config: &HashMap<String, String>) -> Option<String> {
let requested = config.get(PURGE_REQUESTED_KEY).map(|s| s.trim())?;
if requested.is_empty() {
return None;
}
match config.get(PURGE_COMPLETED_KEY) {
Some(completed) if completed.trim() == requested => None,
_ => Some(requested.to_string()),
}
}
/// Records a purge request on the table and returns its token. The next
/// cleanup detection proposes the purge.
pub async fn request_purge(dataset: &mut Dataset) -> Result<String> {
let token = Utc::now().to_rfc3339_opts(chrono::SecondsFormat::Nanos, true);
dataset
.update_config([(PURGE_REQUESTED_KEY, token.as_str())])
.await
.context("record purge request")?;
Ok(token)
}
#[derive(Debug, Default, Clone, PartialEq, Eq)]
pub struct PurgeStats {
pub fragments_rewritten: usize,
pub old_versions: u64,
pub bytes_removed: u64,
/// False while versions from before the rebuild remain.
pub served: bool,
}
/// Rewrites every fragment that has deleted rows and rebuilds the indices from
/// the rows that remain, then removes the versions from before that through
/// the cleanup job's retention policy with only the latest version kept, and
/// marks `token` served once none of them is left. The rebuild is needed
/// because compaction only remaps an index: an IVF_FLAT index keeps the
/// deleted vector's values in its partitions under a remapped row address. It
/// runs once per request, so a purge that has to wait for its cleanup (a
/// version pinned by a tag, a query lease or one someone named, fails it)
/// retries only the cleanup. Files no version references stay while Lance
/// cannot tell them from a writer's uncommitted ones, which is up to seven
/// days for an abandoned write.
pub async fn purge(dataset: &mut Dataset, token: &str) -> Result<PurgeStats> {
let fragments_rewritten = rebuild(dataset, token).await?;
let cleanup = match retention_policy(dataset, 0, 1).await? {
Some(policy) => cleanup_old_versions(dataset, policy)
.await
.context("remove versions older than the purge")?,
None => RemovalStats::default(),
};
let served = finish(dataset, token).await?;
Ok(PurgeStats {
fragments_rewritten,
old_versions: cleanup.old_versions,
bytes_removed: cleanup.bytes_removed,
served,
})
}
/// Rewrites the fragments with deletions and rebuilds the indices for
/// `token`, unless that was done already, and returns how many fragments it
/// rewrote.
pub async fn rebuild(dataset: &mut Dataset, token: &str) -> Result<usize> {
if dataset.config().get(PURGE_REBUILT_KEY).map(|s| s.trim()) == Some(token) {
return Ok(0);
}
let vector_indices = vector_index_names(dataset).await?;
let options = CompactionOptions {
materialize_deletions: true,
// Zero materializes every fragment with any deletion.
materialize_deletions_threshold: 0.0,
..Default::default()
};
let compaction = compact_files(dataset, options, None)
.await
.context("rewrite fragments with deletions")?;
dataset.checkout_latest().await?;
if !vector_indices.is_empty() {
let mut options = OptimizeOptions::retrain();
options.index_names = Some(vector_indices);
dataset
.optimize_indices(&options)
.await
.context("rebuild indices without the deleted rows")?;
}
rebuild_scalar_indices(dataset).await?;
dataset.checkout_latest().await?;
dataset
.update_config([(PURGE_REBUILT_KEY, token)])
.await
.context("record purge rebuild")?;
Ok(compaction.fragments_removed)
}
/// Marks `token` served when the oldest version left is one the rebuild for
/// it wrote, so no version from before the rebuild remains.
pub async fn finish(dataset: &mut Dataset, token: &str) -> Result<bool> {
let oldest = dataset
.versions()
.await?
.iter()
.map(|v| v.version)
.min()
.context("the table has no versions")?;
let oldest = dataset.checkout_version(oldest).await?;
if oldest.config().get(PURGE_REBUILT_KEY).map(|s| s.trim()) != Some(token) {
return Ok(false);
}
dataset
.update_config([(PURGE_COMPLETED_KEY, token)])
.await
.context("record purge completion")?;
Ok(true)
}
/// Names of the vector indices. A retrain builds each name over the whole
/// table, so a column whose index is sharded over several names is refused:
/// its shards would overlap.
async fn vector_index_names(dataset: &Dataset) -> Result<Vec<String>> {
let mut by_column: BTreeMap<Vec<i32>, BTreeSet<String>> = BTreeMap::new();
for index in dataset
.load_indices()
.await?
.iter()
.filter(|i| is_vector(i))
{
by_column
.entry(index.fields.clone())
.or_default()
.insert(index.name.clone());
}
let mut names = Vec::new();
for shards in by_column.into_values() {
if shards.len() > 1 {
bail!(
"the vector index is sharded over {shards:?}; a purge would rebuild each over the whole table"
);
}
names.extend(shards);
}
Ok(names)
}
/// Builds every scalar index again from the rows that remain. Compaction only
/// remaps them: a bitmap index keeps a deleted key, and under stable row ids
/// no scalar index is rewritten at all.
async fn rebuild_scalar_indices(dataset: &mut Dataset) -> Result<()> {
let indices = dataset.load_indices().await?;
let mut rebuilt = BTreeSet::new();
for index in indices.iter() {
if is_system_index(index) || is_vector(index) || !rebuilt.insert(index.name.clone()) {
continue;
}
let [field] = index.fields[..] else {
bail!("index {} covers {} columns", index.name, index.fields.len());
};
let column = dataset.schema().field_path(field)?;
let params = dataset
.open_scalar_index(&column, &index.uuid, &NoOpMetricsCollector)
.await?
.derive_index_params()?;
dataset
.create_index(
&[column.as_str()],
IndexType::Scalar,
Some(index.name.clone()),
&params,
true,
)
.await
.with_context(|| format!("rebuild index {}", index.name))?;
}
Ok(())
}
fn is_vector(index: &IndexMetadata) -> bool {
index
.index_details
.as_ref()
.is_some_and(|details| details.type_url.ends_with("VectorIndexDetails"))
}
#[cfg(test)]
mod tests {
use super::*;
fn config(pairs: &[(&str, &str)]) -> HashMap<String, String> {
pairs
.iter()
.map(|(k, v)| (k.to_string(), v.to_string()))
.collect()
}
const WORKER: Retention = Retention {
retain_hours: 168,
min_versions: 5,
};
#[test]
fn table_retention_overrides_within_range() {
assert_eq!(
Retention::for_table(&config(&[]), WORKER, 8760, 1000),
WORKER
);
assert_eq!(
Retention::for_table(
&config(&[(RETAIN_HOURS_KEY, "0"), (MIN_VERSIONS_KEY, "1")]),
WORKER,
8760,
1000
),
Retention {
retain_hours: 0,
min_versions: 1
}
);
for bad in ["-1", "8761", "x", ""] {
assert_eq!(
Retention::for_table(&config(&[(RETAIN_HOURS_KEY, bad)]), WORKER, 8760, 1000)
.retain_hours,
168,
"{bad:?}"
);
}
assert_eq!(
Retention::for_table(&config(&[(MIN_VERSIONS_KEY, "0")]), WORKER, 8760, 1000)
.min_versions,
5
);
}
#[test]
fn purge_is_pending_until_its_token_is_served() {
assert_eq!(pending(&config(&[])), None);
assert_eq!(pending(&config(&[(PURGE_REQUESTED_KEY, "")])), None);
assert_eq!(
pending(&config(&[(PURGE_REQUESTED_KEY, "t1")])),
Some("t1".to_string())
);
assert_eq!(
pending(&config(&[
(PURGE_REQUESTED_KEY, "t2"),
(PURGE_COMPLETED_KEY, "t1")
])),
Some("t2".to_string())
);
assert_eq!(
pending(&config(&[
(PURGE_REQUESTED_KEY, "t2"),
(PURGE_COMPLETED_KEY, "t2")
])),
None
);
}
}
+8
View File
@@ -5,12 +5,18 @@
//! and its old versions removed. None can run in the Go worker, because all
//! three read and rewrite Lance files. This crate is the worker that can.
// A job awaiting several Lance commits in turn needs deeper trait recursion
// than the default to prove its future Send.
#![recursion_limit = "256"]
pub mod catalog;
pub mod cluster;
pub mod dataset;
pub mod direct_read;
pub mod drop_guard;
pub mod grpc_tls;
pub mod jobs;
pub mod metadata_index;
pub mod metrics;
pub mod preview;
pub mod security;
@@ -30,3 +36,5 @@ pub mod volume_server_pb {
pub mod remote_pb {
tonic::include_proto!("remote_pb");
}
#[cfg(test)]
mod drop_tests;
File diff suppressed because it is too large. Load diff
+394
View File
@@ -0,0 +1,394 @@
//! A purge removes a deleted vector's bytes from storage: after it, no file
//! under the table holds the vector, its key or its metadata. Runs on a local
//! table, so it needs no cluster.
use std::path::Path;
use std::sync::Arc;
use arrow_array::types::Float32Type;
use arrow_array::{FixedSizeListArray, RecordBatch, RecordBatchIterator, StringArray};
use arrow_schema::{DataType, Field, Schema};
use lance::Dataset;
use lance::dataset::cleanup::{CleanupPolicy, RemovalStats, cleanup_old_versions};
use lance::index::DatasetIndexExt;
use lance::index::vector::VectorIndexParams;
use lance_index::IndexType;
use lance_linalg::distance::DistanceType;
use weed_lance_worker::jobs::purge;
const DIM: i32 = 8;
const ROWS: usize = 300;
const MARKER_ROW: usize = 137;
const KEY_MARKER: &str = "purge-key-5f0c2e9a7d41b3c8";
const METADATA_MARKER: &str = "purge-metadata-a93e1d5b77c2f046";
fn marker_vector() -> Vec<f32> {
// Values no other row has, so their bytes identify the row.
(0..DIM).map(|i| 31_337.125 + i as f32 * 0.5).collect()
}
fn batch() -> (Arc<Schema>, RecordBatch) {
let schema = Arc::new(Schema::new(vec![
Field::new("key", DataType::Utf8, false),
Field::new(
"vector",
DataType::FixedSizeList(Arc::new(Field::new("item", DataType::Float32, true)), DIM),
false,
),
Field::new("metadata", DataType::Utf8, true),
]));
let keys: Vec<String> = (0..ROWS)
.map(|i| {
if i == MARKER_ROW {
KEY_MARKER.to_string()
} else {
format!("k{i}")
}
})
.collect();
let metadata: Vec<String> = (0..ROWS)
.map(|i| {
if i == MARKER_ROW {
format!("{{\"note\":\"{METADATA_MARKER}\"}}")
} else {
format!("{{\"n\":{i}}}")
}
})
.collect();
let vectors = FixedSizeListArray::from_iter_primitive::<Float32Type, _, _>(
(0..ROWS).map(|i| {
let values: Vec<f32> = if i == MARKER_ROW {
marker_vector()
} else {
(0..DIM)
.map(|d| ((i * 7 + d as usize) % 97) as f32)
.collect()
};
Some(values.into_iter().map(Some).collect::<Vec<_>>())
}),
DIM,
);
let batch = RecordBatch::try_new(
schema.clone(),
vec![
Arc::new(StringArray::from(keys)),
Arc::new(vectors),
Arc::new(StringArray::from(metadata)),
],
)
.unwrap();
(schema, batch)
}
fn vector_bytes() -> Vec<u8> {
marker_vector()
.iter()
.flat_map(|v| v.to_le_bytes())
.collect()
}
/// Each marker found under `dir`, as "<marker> in <file>".
fn markers_in(dir: &Path) -> Vec<String> {
let vector = vector_bytes();
let mut found = Vec::new();
let mut stack = vec![dir.to_path_buf()];
while let Some(path) = stack.pop() {
for entry in std::fs::read_dir(&path).unwrap() {
let entry = entry.unwrap();
if entry.file_type().unwrap().is_dir() {
stack.push(entry.path());
continue;
}
let bytes = std::fs::read(entry.path()).unwrap();
let has = |needle: &[u8]| bytes.windows(needle.len()).any(|w| w == needle);
for (name, needle) in [
("vector", vector.as_slice()),
("key", KEY_MARKER.as_bytes()),
("metadata", METADATA_MARKER.as_bytes()),
] {
if has(needle) {
let file = entry.path();
let file = file.strip_prefix(dir).unwrap_or(&file);
found.push(format!("{name} in {}", file.display()));
}
}
}
}
found.sort_unstable();
found
}
/// A purge whose cleanup nothing holds back, so only the latest version
/// stays. [`purge::purge`] cleans up through the cleanup job's retention
/// policy, which a deployment may have keep more.
async fn purge_to_latest(
dataset: &mut Dataset,
token: &str,
) -> anyhow::Result<(usize, RemovalStats, bool)> {
let rewritten = purge::rebuild(dataset, token).await?;
let policy = CleanupPolicy {
before_timestamp: Some(chrono::Utc::now()),
before_version: Some(dataset.latest_version_id().await?),
error_if_tagged_old_versions: true,
..Default::default()
};
let removed = cleanup_old_versions(dataset, policy).await?;
Ok((rewritten, removed, purge::finish(dataset, token).await?))
}
#[tokio::test]
async fn purge_removes_a_deleted_vectors_bytes() {
purge_leaves_no_trace(false).await;
}
/// Under stable row ids compaction leaves scalar indices as they are, so only
/// a rebuild takes the key out of them.
#[tokio::test]
async fn purge_removes_a_deleted_vectors_bytes_under_stable_row_ids() {
purge_leaves_no_trace(true).await;
}
async fn purge_leaves_no_trace(stable_row_ids: bool) {
let dir = tempfile::tempdir().unwrap();
let uri = dir.path().join("index.lance");
let uri = uri.to_str().unwrap();
let (schema, batch) = batch();
let params = lance::dataset::WriteParams {
enable_stable_row_ids: stable_row_ids,
..Default::default()
};
let mut dataset = Dataset::write(
RecordBatchIterator::new(vec![Ok(batch)], schema),
uri,
Some(params),
)
.await
.unwrap();
dataset
.create_index(
&["vector"],
IndexType::Vector,
Some("vector_idx".to_string()),
&VectorIndexParams::ivf_flat(2, DistanceType::L2),
false,
)
.await
.unwrap();
dataset
.create_index(
&["key"],
IndexType::BTree,
Some("key_idx".to_string()),
&lance_index::scalar::ScalarIndexParams::default(),
false,
)
.await
.unwrap();
// A remapped bitmap index still lists the deleted row's value.
dataset
.create_index(
&["metadata"],
IndexType::Bitmap,
Some("metadata_idx".to_string()),
&lance_index::scalar::ScalarIndexParams::for_builtin(
lance_index::scalar::BuiltinIndexType::Bitmap,
),
false,
)
.await
.unwrap();
dataset.checkout_latest().await.unwrap();
let before = markers_in(dir.path());
assert!(
before.iter().any(|m| m.starts_with("vector ")),
"the vector's bytes should be found before deletion, found {before:?}"
);
dataset
.delete(&format!("key = '{KEY_MARKER}'"))
.await
.unwrap();
assert_eq!(dataset.count_rows(None).await.unwrap(), ROWS - 1);
// A deletion alone, and the default retention cleanup, leave the bytes:
// the fragment still holds the row and older versions still reference it.
cleanup_old_versions(
&dataset,
CleanupPolicy {
before_timestamp: Some(chrono::Utc::now() - chrono::Duration::hours(168)),
..Default::default()
},
)
.await
.unwrap();
assert!(
markers_in(dir.path())
.iter()
.any(|m| m.starts_with("vector ")),
"a deleted vector is expected to stay until purged"
);
let token = purge::request_purge(&mut dataset).await.unwrap();
assert_eq!(purge::pending(dataset.config()), Some(token.clone()));
let (rewritten, removed, served) = purge_to_latest(&mut dataset, &token).await.unwrap();
assert!(served);
assert!(rewritten >= 1);
assert!(removed.old_versions >= 1, "{removed:?}");
assert_eq!(
markers_in(dir.path()),
Vec::<String>::new(),
"deleted vector's bytes remain in storage after the purge"
);
assert_eq!(purge::pending(dataset.config()), None);
// The table is intact: every other row, and the index, still serve.
let reopened = Dataset::open(uri).await.unwrap();
assert_eq!(reopened.count_rows(None).await.unwrap(), ROWS - 1);
assert_eq!(purge::pending(reopened.config()), None);
assert_eq!(reopened.load_indices().await.unwrap().len(), 3);
}
/// A version a query lease (a tag) pins still holds the deleted row, so the
/// purge must fail and leave the request pending rather than report success.
/// Once the tag goes, the next purge only cleans up: the rebuild is not done
/// again.
#[tokio::test]
async fn purge_refuses_while_a_tag_pins_old_data() {
let dir = tempfile::tempdir().unwrap();
let uri = dir.path().join("index.lance");
let uri = uri.to_str().unwrap();
let (schema, batch) = batch();
let mut dataset = Dataset::write(RecordBatchIterator::new(vec![Ok(batch)], schema), uri, None)
.await
.unwrap();
dataset
.create_index(
&["vector"],
IndexType::Vector,
Some("vector_idx".to_string()),
&VectorIndexParams::ivf_flat(2, DistanceType::L2),
false,
)
.await
.unwrap();
let pinned = dataset.version().version;
dataset
.tags()
.create(
"weed-vector-query-lease",
lance::dataset::refs::Ref::Version(None, Some(pinned)),
)
.await
.unwrap();
dataset
.delete(&format!("key = '{KEY_MARKER}'"))
.await
.unwrap();
let token = purge::request_purge(&mut dataset).await.unwrap();
assert!(purge_to_latest(&mut dataset, &token).await.is_err());
dataset.checkout_latest().await.unwrap();
assert_eq!(purge::pending(dataset.config()), Some(token.clone()));
let index = |dataset: &Dataset| {
let dataset = dataset.clone();
async move { dataset.load_indices().await.unwrap()[0].uuid }
};
let rebuilt = index(&dataset).await;
dataset
.tags()
.delete("weed-vector-query-lease")
.await
.unwrap();
let (rewritten, _, served) = purge_to_latest(&mut dataset, &token).await.unwrap();
assert!(served);
assert_eq!(rewritten, 0);
assert_eq!(
index(&dataset).await,
rebuilt,
"the index was rebuilt again"
);
assert_eq!(purge::pending(dataset.config()), None);
assert_eq!(markers_in(dir.path()), Vec::<String>::new());
}
/// The request is served only once no version from before the rebuild is
/// left, whatever kept one.
#[tokio::test]
async fn purge_is_served_once_older_versions_are_gone() {
let dir = tempfile::tempdir().unwrap();
let uri = dir.path().join("index.lance");
let uri = uri.to_str().unwrap();
let (schema, batch) = batch();
let mut dataset = Dataset::write(RecordBatchIterator::new(vec![Ok(batch)], schema), uri, None)
.await
.unwrap();
dataset
.delete(&format!("key = '{KEY_MARKER}'"))
.await
.unwrap();
let token = purge::request_purge(&mut dataset).await.unwrap();
assert_eq!(purge::rebuild(&mut dataset, &token).await.unwrap(), 1);
assert!(!purge::finish(&mut dataset, &token).await.unwrap());
assert_eq!(purge::pending(dataset.config()), Some(token.clone()));
let (rewritten, removed, served) = purge_to_latest(&mut dataset, &token).await.unwrap();
assert!(served);
assert_eq!(rewritten, 0);
assert!(removed.old_versions >= 1, "{removed:?}");
assert_eq!(purge::pending(dataset.config()), None);
}
/// A retrain builds every index name over the whole table, so shards of one
/// vector index would end up overlapping: the purge refuses before touching
/// the table and leaves the request pending.
#[tokio::test]
async fn purge_refuses_a_sharded_vector_index() {
let dir = tempfile::tempdir().unwrap();
let uri = dir.path().join("index.lance");
let uri = uri.to_str().unwrap();
let (schema, batch) = batch();
let params = lance::dataset::WriteParams {
max_rows_per_file: ROWS / 2,
..Default::default()
};
let mut dataset = Dataset::write(
RecordBatchIterator::new(vec![Ok(batch)], schema),
uri,
Some(params),
)
.await
.unwrap();
let fragments: Vec<u32> = dataset
.get_fragments()
.iter()
.map(|f| f.id() as u32)
.collect();
assert_eq!(fragments.len(), 2);
for (shard, fragment) in fragments.iter().enumerate() {
dataset
.create_index_builder(
&["vector"],
IndexType::Vector,
&VectorIndexParams::ivf_flat(2, DistanceType::L2),
)
.name(format!("shard-{shard}"))
.fragments(vec![*fragment])
.await
.unwrap();
dataset.checkout_latest().await.unwrap();
}
dataset
.delete(&format!("key = '{KEY_MARKER}'"))
.await
.unwrap();
let token = purge::request_purge(&mut dataset).await.unwrap();
let version = dataset.version().version;
let err = purge::purge(&mut dataset, &token).await.unwrap_err();
assert!(format!("{err:#}").contains("sharded"), "{err:#}");
dataset.checkout_latest().await.unwrap();
assert_eq!(dataset.version().version, version, "the table was changed");
assert_eq!(purge::pending(dataset.config()), Some(token));
}
+3 -3
View File
@@ -13,9 +13,9 @@ require (
github.com/beorn7/perks v1.0.1 // indirect
github.com/cespare/xxhash/v2 v2.3.0 // indirect
github.com/munnerz/goautoneg v0.0.0-20191010083416-a7dc8b61c822 // indirect
github.com/prometheus/common v0.70.1 // indirect
github.com/prometheus/procfs v0.22.0 // indirect
golang.org/x/sys v0.48.0 // indirect
github.com/prometheus/common v0.71.0 // indirect
github.com/prometheus/procfs v0.23.0 // indirect
golang.org/x/sys v0.49.0 // indirect
)
replace github.com/seaweedfs/seaweedfs => ../..
+12 -16
View File
@@ -2,35 +2,31 @@ github.com/beorn7/perks v1.0.1 h1:VlbKKnNfV8bJzeqoa4cOKqO6bYr3WgKZxO8Z16+hsOM=
github.com/beorn7/perks v1.0.1/go.mod h1:G2ZrVWU2WbWT9wwq4/hrbKbnv/1ERSJQ0ibhJ6rlkpw=
github.com/cespare/xxhash/v2 v2.3.0 h1:UL815xU9SqsFlibzuggzjXhog7bL6oX9BbNZnL2UFvs=
github.com/cespare/xxhash/v2 v2.3.0/go.mod h1:VGX0DQ3Q6kWi7AoAeZDth3/j3BFtOZR5XLFGgcrjCOs=
github.com/davecgh/go-spew v1.1.2-0.20180830191138-d8f796af33cc h1:U9qPSI2PIWSS1VwoXQT9A3Wy9MM3WgvqSxFWenqJduM=
github.com/davecgh/go-spew v1.1.2-0.20180830191138-d8f796af33cc/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
github.com/google/go-cmp v0.7.0 h1:wk8382ETsv4JYUZwIsn6YpYiWiBsYLSJiTsyBybVuN8=
github.com/google/go-cmp v0.7.0/go.mod h1:pXiqmnSA92OHEEa9HXL2W4E7lf9JzCmGVUdgjX3N/iU=
github.com/klauspost/compress v1.19.2 h1:hMRETovs/pu/dVWN7zIT1PGG8t509MwT6bO7XSi26R8=
github.com/klauspost/compress v1.19.2/go.mod h1:cwPg85FWrGar70rWktvGQj8/hthj3wpl0PGDogxkrSQ=
github.com/klauspost/compress v1.20.3 h1:zWNoFqq/lSuYQRim3dUXlXeA0IqwuZGicdG5ZTfY5j4=
github.com/klauspost/compress v1.20.3/go.mod h1:LUdAzn7YLVvxLpc7y3V1m40wESHTgc1422pwwBSKYuI=
github.com/kylelemons/godebug v1.1.0 h1:RPNrshWIDI6G2gRW9EHilWtl7Z6Sb1BR0xunSBf0SNc=
github.com/kylelemons/godebug v1.1.0/go.mod h1:9/0rRGxNHcop5bhtWyNeEfOS8JIWk580+fNqagV/RAw=
github.com/munnerz/goautoneg v0.0.0-20191010083416-a7dc8b61c822 h1:C3w9PqII01/Oq1c1nUAm88MOHcQC9l5mIlSMApZMrHA=
github.com/munnerz/goautoneg v0.0.0-20191010083416-a7dc8b61c822/go.mod h1:+n7T8mK8HuQTcFwEeznm/DIxMOiR9yIdICNftLE1DvQ=
github.com/pmezard/go-difflib v1.0.1-0.20181226105442-5d4384ee4fb2 h1:Jamvg5psRIccs7FGNTlIRMkT8wgtp5eCXdBlqhYGL6U=
github.com/pmezard/go-difflib v1.0.1-0.20181226105442-5d4384ee4fb2/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4=
github.com/prometheus/client_golang v1.24.1 h1:JnJkREXzWxUdCuPFpIWZiPispT9xVV59uiuyR2bPlnU=
github.com/prometheus/client_golang v1.24.1/go.mod h1:F+oSRECHg4sse5ucfYpYDeIv/hu68Zo0uoHKetWnzcE=
github.com/prometheus/client_model v0.6.3 h1:O0jaTVAYNxTHYInEPFJt5I3+sN8zqBtVMPTB1qyxiEo=
github.com/prometheus/client_model v0.6.3/go.mod h1:gpN5P9S7Rr6Yr92PiQ+Ixvhf6JZEkF1dnxsYL2aPBEM=
github.com/prometheus/common v0.70.1 h1:1HvjP4D5oL3t8RsPlwxA9onvvStjtIHYE5XuuwOi/PY=
github.com/prometheus/common v0.70.1/go.mod h1:VdFUQDMZK3VLkurFUVhia6uys/0suUp86TJz5qbJRhc=
github.com/prometheus/procfs v0.22.0 h1:6q9+/JL9IKAPbCmBrv9n5O5Ty3NKnciV5X7YGw0oics=
github.com/prometheus/procfs v0.22.0/go.mod h1:CvmFr/GVhIjIvWJZW3tgkODBQMRIf0EyWMQLHCHab58=
github.com/stretchr/testify v1.11.1 h1:7s2iGBzp5EwR7/aIZr8ao5+dra3wiQyKjjFuvgVKu7U=
github.com/stretchr/testify v1.11.1/go.mod h1:wZwfW3scLgRK+23gO65QZefKpKQRnfz6sD981Nm4B6U=
github.com/prometheus/common v0.71.0 h1:9KDAKb7Mj3HEVKyFCK6Dc/HIwlBzZIN2l7/lrHl3KK8=
github.com/prometheus/common v0.71.0/go.mod h1:CLJ5H8TEsGX8bl31BdMkfhIZ+QmZ9tBPPotUxUbfcmk=
github.com/prometheus/procfs v0.23.0 h1:ad+S6FTkzwaMgacKLM5h6bNKExS+DMH0YiUwXItH/o4=
github.com/prometheus/procfs v0.23.0/go.mod h1:3AVXO0FqLafyYFU/V/IQRZlC8Rgqz8Dm/x6pKLK+RIk=
github.com/stretchr/testify v1.12.1 h1:EuwCh5fleGS7H32xRwO3wRGT7DxrDhLAT6FF8MpWDWE=
github.com/stretchr/testify v1.12.1/go.mod h1:MDEgiDPPsNp5cuIrHPPCyornHKgEVbtFUmoNlxoYthg=
go.uber.org/goleak v1.3.0 h1:2K3zAYmnTNqV73imy9J1T3WC+gmCePx2hEGkimedGto=
go.uber.org/goleak v1.3.0/go.mod h1:CoHD4mav9JJNrW/WLlf7HGZPjdw8EucARQHekz1X6bE=
go.yaml.in/yaml/v2 v2.4.4 h1:tuyd0P+2Ont/d6e2rl3be67goVK4R6deVxCUX5vyPaQ=
go.yaml.in/yaml/v2 v2.4.4/go.mod h1:gMZqIpDtDqOfM0uNfy0SkpRhvUryYH0Z6wdMYcacYXQ=
golang.org/x/sys v0.48.0 h1:bbX/i/6MgT9BVLM9RT1thmxL04yeTAhbEz4SyadbXoo=
golang.org/x/sys v0.48.0/go.mod h1:hNLxWAXmnKAxqDtdwIYC4bM9oQPEecfsnNMuSxOs3og=
go.yaml.in/yaml/v3 v3.0.6 h1:zcDfvjXEqTQmpjniUBLc91xYJMB2c4QOFxEF/LCEckM=
go.yaml.in/yaml/v3 v3.0.6/go.mod h1:HVTZu1O7/Vkt2N+BFy8Zza+lnLsABggaTM2ZpNIGuKg=
golang.org/x/sys v0.49.0 h1:XbzkgYJHdqh/8m2Uu0W/dQv8nktxx4BFHp1M0gROTXA=
golang.org/x/sys v0.49.0/go.mod h1:hNLxWAXmnKAxqDtdwIYC4bM9oQPEecfsnNMuSxOs3og=
google.golang.org/protobuf v1.36.12 h1:pJOKDDOyeXErUroCihFAd5LQuwXBSpVnKGrj5o/fwxc=
google.golang.org/protobuf v1.36.12/go.mod h1:HTf+CrKn2C3g5S8VImy6tdcUvCska2kB7j23XfzDpco=
gopkg.in/yaml.v3 v3.0.1 h1:fxVm/GzAzEWqLHuvctI91KS9hhNmmWOoWu0XTYJS7CA=
gopkg.in/yaml.v3 v3.0.1/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM=
+22 -1
View File
@@ -74,9 +74,15 @@ func (s *PrometheusStorage) GetVersionSeries(days, limit int) VersionSeries {
if limit > 0 && len(series.Versions) > limit {
// Old releases are a long tail of one-cluster bands; keep the versions
// most of the fleet is on now and sum the rest into one band.
// most of the fleet is on now and sum the rest into one band. The
// newest release is kept too, while it is still too small to rank, or
// clusters upgrading to it look like they left the named versions.
newest := newestRelease(series.Versions, last)
ranked := append([]VersionCounts(nil), series.Versions...)
sort.Slice(ranked, func(i, j int) bool {
if (ranked[i].Version == newest) != (ranked[j].Version == newest) {
return ranked[i].Version == newest
}
if ranked[i].Clusters[last] != ranked[j].Clusters[last] {
return ranked[i].Clusters[last] > ranked[j].Clusters[last]
}
@@ -104,6 +110,21 @@ func (s *PrometheusStorage) GetVersionSeries(days, limit int) VersionSeries {
return series
}
// newestRelease is the newest plain release, without a build suffix, that
// clusters run on the last day.
func newestRelease(versions []VersionCounts, last int) string {
newest := ""
for _, v := range versions {
if _, suffix, ok := versionParts(v.Version); !ok || suffix != "" || v.Clusters[last] == 0 {
continue
}
if newest == "" || versionLess(newest, v.Version) {
newest = v.Version
}
}
return newest
}
// versionedHistories drops the samples recorded before the reported version was
// kept in history, so the version series spans the days it actually knows a
// version for instead of climbing out of a run of blank days.
+26
View File
@@ -93,6 +93,32 @@ func TestVersionSeriesHoldsForwardAndLimits(t *testing.T) {
}
}
func TestVersionSeriesKeepsNewestRelease(t *testing.T) {
s := newPrometheusStorage(prometheus.NewRegistry())
seedSamples(s, "a", HistorySample{Version: "4.40"}, -2, -1, 0)
seedSamples(s, "b", HistorySample{Version: "4.40"}, -2, -1, 0)
seedSamples(s, "c", HistorySample{Version: "4.41"}, -2, -1, 0)
seedSamples(s, "d", HistorySample{Version: "4.41"}, -2, -1, 0)
// Upgraded to the newest release, which is still the smallest band.
seedSamples(s, "e", HistorySample{Version: "4.41"}, -2)
seedSamples(s, "e", HistorySample{Version: "4.42"}, -1, 0)
// A suffixed build newer than every release does not take its place.
seedSamples(s, "f", HistorySample{Version: "4.43-dev"}, -2, -1, 0)
series := s.GetVersionSeries(3, 2)
var kept []string
for _, v := range series.Versions {
kept = append(kept, v.Version)
}
if len(kept) != 2 || kept[0] != "4.41" || kept[1] != "4.42" {
t.Fatalf("kept = %v, want 4.41 and the newest release 4.42", kept)
}
if series.Other == nil || series.Other.Count != 2 {
t.Fatalf("other = %+v, want 4.40 and 4.43-dev", series.Other)
}
}
func TestVersionLess(t *testing.T) {
// Ordered oldest to newest; every pair must compare in this order.
ordered := []string{"unknown", "3.97", "4.02", "4.30", "4.30-enterprise", "9.99", "10.02"}
+14 -1
View File
@@ -29,6 +29,7 @@ type FuseTestFramework struct {
mountProcess *managedProcess
filerAddr string
filerPort int
masterPort int
weedBinary string
isSetup bool
config *TestConfig
@@ -74,6 +75,10 @@ func NewFuseTestFramework(t *testing.T, config *TestConfig) *FuseTestFramework {
require.NoError(t, err)
filerPort := freePort(t)
masterPort := freePort(t)
for masterPort == filerPort {
masterPort = freePort(t)
}
return &FuseTestFramework{
t: t,
@@ -83,6 +88,7 @@ func NewFuseTestFramework(t *testing.T, config *TestConfig) *FuseTestFramework {
logDir: filepath.Join(tempDir, "logs"),
filerPort: filerPort,
filerAddr: fmt.Sprintf("127.0.0.1:%d", filerPort),
masterPort: masterPort,
weedBinary: findWeedBinary(),
isSetup: false,
config: config,
@@ -218,6 +224,11 @@ func (f *FuseTestFramework) GetFilerAddr() string {
return f.filerAddr
}
// GetMasterAddr returns the master HTTP address of the mini cluster.
func (f *FuseTestFramework) GetMasterAddr() string {
return fmt.Sprintf("127.0.0.1:%d", f.masterPort)
}
// managedProcess is a started weed sub-command whose exit is watched, so a wait
// for it to come up ends the moment it dies instead of burning its full timeout.
type managedProcess struct {
@@ -321,7 +332,7 @@ func (f *FuseTestFramework) startMini(config *TestConfig) error {
args = append(args, "-v=4")
}
adminPort := freePort(f.t)
for adminPort == f.filerPort {
for adminPort == f.filerPort || adminPort == f.masterPort {
adminPort = freePort(f.t)
}
args = append(args,
@@ -330,6 +341,8 @@ func (f *FuseTestFramework) startMini(config *TestConfig) error {
"-ip=127.0.0.1",
"-ip.bind=127.0.0.1",
"-filer.port="+strconv.Itoa(f.filerPort),
"-master.port="+strconv.Itoa(f.masterPort),
"-master.port.grpc="+strconv.Itoa(f.masterPort+grpcPortOffset),
"-s3=false",
"-webdav=false",
"-admin.ui=false",
@@ -0,0 +1,214 @@
//go:build linux
package fuse_test
import (
"encoding/binary"
"encoding/json"
"fmt"
"net/http"
"net/url"
"os"
"path/filepath"
"strings"
"testing"
"time"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
"golang.org/x/sys/unix"
)
const (
mergePageSize = 8192
mergePages = 32
)
// mergeHTTP bounds every HTTP call the test makes to the cluster.
var mergeHTTP = &http.Client{Timeout: 30 * time.Second}
// mergeVacuumHTTP outlasts a master vacuum, which waits up to 30 s per volume
// for writes assigned to it to drain before compacting it.
var mergeVacuumHTTP = &http.Client{Timeout: 3 * time.Minute}
// TestReadOpenHandleAfterChunkMerge covers a file a flush merged while a
// descriptor kept it open. The merge rewrites the file into new chunks and the
// filer deletes the chunks it replaced, so after a vacuum only the new chunks
// exist. Reads through the open handle must find them at once, whether or not
// the handle has a write it has not flushed yet.
func TestReadOpenHandleAfterChunkMerge(t *testing.T) {
config := DefaultTestConfig()
// The client chunk cache could still hold a replaced chunk; the reads
// below have to reach the volume server.
config.CacheSizeMB = 0
config.MountOptions = append(config.MountOptions, "-cacheCapacityMB=0")
framework := NewFuseTestFramework(t, config)
defer framework.Cleanup()
require.NoError(t, framework.Setup(config))
t.Run("CleanHandle", func(t *testing.T) {
testReadOpenHandleAfterChunkMerge(t, framework, "merged_clean.dat", false)
})
t.Run("HandleWithUnflushedWrite", func(t *testing.T) {
testReadOpenHandleAfterChunkMerge(t, framework, "merged_dirty.dat", true)
})
}
func testReadOpenHandleAfterChunkMerge(t *testing.T, framework *FuseTestFramework, name string, unflushedWrite bool) {
path := filepath.Join(framework.GetMountPoint(), name)
held, err := os.OpenFile(path, os.O_CREATE|os.O_RDWR|os.O_TRUNC, 0644)
require.NoError(t, err)
defer held.Close()
version := make([]int, mergePages)
whole := make([]byte, 0, mergePages*mergePageSize)
for p := 0; p < mergePages; p++ {
whole = append(whole, mergePagePattern(p, 0)...)
}
_, err = held.WriteAt(whole, 0)
require.NoError(t, err)
require.NoError(t, held.Sync())
// Overlapping 16-page windows: each leaves part of every older window
// visible, so the stored chunks soon add up to more than twice the file
// size and the next flush merges them.
var replaced []mergeChunk
for i, start := range []int{0, 3, 6, 9, 12, 15} {
before := mergeEntryChunks(t, framework, name)
window := make([]byte, 0, 16*mergePageSize)
for p := start; p < start+16; p++ {
window = append(window, mergePagePattern(p, i+1)...)
version[p] = i + 1
}
_, err = held.WriteAt(window, int64(start*mergePageSize))
require.NoError(t, err)
require.NoError(t, held.Sync())
if after := mergeEntryChunks(t, framework, name); len(after) == 1 && after[0].ModifiedTsNs == 0 {
replaced = before
break
}
}
require.NotEmpty(t, replaced, "the overlapping writes never made a flush merge the chunks")
// The filer deletes the replaced chunks after the merge commits; vacuum
// once they are deleted so that only the merged chunk is left on disk.
master := framework.GetMasterAddr()
require.Eventually(t, func() bool {
for _, c := range replaced {
if mergeChunkStatus(master, c, false) != http.StatusNotFound {
return false
}
}
return true
}, 30*time.Second, 200*time.Millisecond, "the replaced chunks were never deleted")
require.Eventually(t, func() bool {
resp, err := mergeVacuumHTTP.Get(fmt.Sprintf("http://%s/vol/vacuum?garbageThreshold=0.0001", master))
if err != nil {
return false
}
resp.Body.Close()
if resp.StatusCode != http.StatusOK {
return false
}
for _, c := range replaced {
if mergeChunkStatus(master, c, true) != http.StatusNotFound {
return false
}
}
return true
}, 4*time.Minute, time.Second, "vacuum left the replaced chunks on disk")
if unflushedWrite {
_, err = held.WriteAt(mergePagePattern(mergePages-1, 99), int64((mergePages-1)*mergePageSize))
require.NoError(t, err)
version[mergePages-1] = 99
}
// A second descriptor shares the handle the first one keeps open. Drop its
// pages from the kernel cache so every read goes to the mount.
reader, err := os.Open(path)
require.NoError(t, err)
defer reader.Close()
require.NoError(t, unix.Fadvise(int(reader.Fd()), 0, 0, unix.FADV_DONTNEED))
started := time.Now()
buf := make([]byte, mergePageSize)
for p := 0; p < mergePages; p++ {
n, err := reader.ReadAt(buf, int64(p*mergePageSize))
require.NoError(t, err, "read page %d through the open handle after the merge", p)
require.Equal(t, mergePageSize, n, "page %d", p)
require.Equal(t, mergePagePattern(p, version[p]), buf, "page %d", p)
}
assert.Less(t, time.Since(started), 5*time.Second, "reads through the open handle waited on chunks the merge replaced")
}
func mergePagePattern(page, version int) []byte {
b := make([]byte, mergePageSize)
for i := 0; i < len(b); i += 8 {
binary.LittleEndian.PutUint32(b[i:], uint32(page))
binary.LittleEndian.PutUint32(b[i+4:], uint32(version))
}
return b
}
type mergeChunk struct {
FileId string `json:"file_id"`
ModifiedTsNs int64 `json:"modified_ts_ns"`
Fid *struct {
VolumeId uint32 `json:"volume_id"`
FileKey uint64 `json:"file_key"`
Cookie uint32 `json:"cookie"`
} `json:"fid"`
}
func (c mergeChunk) fileId() string {
if c.FileId != "" || c.Fid == nil {
return c.FileId
}
return fmt.Sprintf("%d,%x%08x", c.Fid.VolumeId, c.Fid.FileKey, c.Fid.Cookie)
}
// mergeEntryChunks returns the chunks the filer stores for the file.
func mergeEntryChunks(t *testing.T, framework *FuseTestFramework, name string) []mergeChunk {
t.Helper()
resp, err := mergeHTTP.Get(fmt.Sprintf("http://%s/%s?metadata=true", framework.GetFilerAddr(), url.PathEscape(name)))
require.NoError(t, err)
defer resp.Body.Close()
require.Equal(t, http.StatusOK, resp.StatusCode, "filer metadata of %s", name)
var entry struct {
Chunks []mergeChunk `json:"chunks"`
}
require.NoError(t, json.NewDecoder(resp.Body).Decode(&entry))
return entry.Chunks
}
// mergeChunkStatus HEADs a chunk on the volume server holding it, and returns
// 0 when it cannot ask. With readDeleted, a deleted needle still answers until
// a vacuum removes it.
func mergeChunkStatus(master string, c mergeChunk, readDeleted bool) int {
fid := c.fileId()
volumeId := strings.SplitN(fid, ",", 2)[0]
resp, err := mergeHTTP.Get(fmt.Sprintf("http://%s/dir/lookup?volumeId=%s", master, volumeId))
if err != nil {
return 0
}
var lookup struct {
Locations []struct {
Url string `json:"url"`
} `json:"locations"`
}
err = json.NewDecoder(resp.Body).Decode(&lookup)
resp.Body.Close()
if err != nil || len(lookup.Locations) == 0 {
return 0
}
target := fmt.Sprintf("http://%s/%s", lookup.Locations[0].Url, fid)
if readDeleted {
target += "?readDeleted=true"
}
head, err := mergeHTTP.Head(target)
if err != nil {
return 0
}
head.Body.Close()
return head.StatusCode
}
@@ -0,0 +1,116 @@
//go:build !windows
package metadata_subscribe
import (
"context"
"fmt"
"os"
"path/filepath"
"strconv"
"testing"
"time"
"github.com/stretchr/testify/require"
"github.com/seaweedfs/seaweedfs/test/testutil"
"github.com/seaweedfs/seaweedfs/weed/pb"
"github.com/seaweedfs/seaweedfs/weed/pb/filer_pb"
)
// A peer removal is broadcast only to the filers connected to the master that
// saw the peer go. A peer that dies while no master is up (or across a leader
// change) is never removed: the filer keeps retrying its address, the peer's
// delivery watermark stays at zero, and every aggregated subscriber of that
// filer is held at the settled horizon (2 x LogFlushInterval) for good.
func TestAggregatedSubscriberNotHeldByPeerThatDiedUnnoticed(t *testing.T) {
if testing.Short() {
t.Skip("skipping integration test in short mode")
}
weedBinary := findWeedBinary()
require.NotEmpty(t, weedBinary, "weed binary not found")
testDir, err := os.MkdirTemp("", "seaweedfs_peer_unreachable_")
require.NoError(t, err)
t.Cleanup(func() {
if t.Failed() {
t.Logf("logs kept at %s", testDir)
return
}
os.RemoveAll(testDir)
})
ports, err := testutil.AllocateMiniPorts(3)
require.NoError(t, err)
masterPort, filer1Port, filer2Port := ports[0], ports[1], ports[2]
master := pb.ServerAddress(fmt.Sprintf("127.0.0.1:%d", masterPort))
filer1Address := fmt.Sprintf("127.0.0.1:%d", filer1Port)
masterArgs := []string{"master",
"-ip=127.0.0.1",
"-port=" + strconv.Itoa(masterPort),
"-mdir=" + mkdir(t, testDir, "master"),
"-peers=none"}
masterProcess := startProcess(t, weedBinary, filepath.Join(testDir, "master.log"), masterArgs...)
require.NoError(t, waitForLeader(masterPort, 60*time.Second))
startFiler(t, weedBinary, testDir, "filer1", filer1Port, masterPort)
require.NoError(t, waitForHTTPServer(fmt.Sprintf("http://127.0.0.1:%d/", filer1Port), 30*time.Second))
filer2 := startFiler(t, weedBinary, testDir, "filer2", filer2Port, masterPort)
require.NoError(t, waitForHTTPServer(fmt.Sprintf("http://127.0.0.1:%d/", filer2Port), 30*time.Second))
filer2Address := fmt.Sprintf("127.0.0.1:%d", filer2Port)
// filer1 follows filer2: an entry created on filer2 reaches filer1
createPeerEntry(t, filer2Address, "from-filer2")
require.NoError(t, waitForPeerEntry(filer1Address, "from-filer2", 60*time.Second),
"filer1 never followed filer2")
ctx, cancel := context.WithCancel(context.Background())
defer cancel()
events := make(chan *filer_pb.SubscribeMetadataResponse, 1000)
sinceNs := time.Now().UnixNano()
go subscribeToMetadataWithOptions(ctx, filer1Address, peerEntryDir, sinceNs, events)
arrived := make(map[string]time.Time)
waitArrival := func(name string, within time.Duration) bool {
deadline := time.After(within)
for {
if _, ok := arrived[name]; ok {
return true
}
select {
case ev := <-events:
if n := ev.GetEventNotification().GetNewEntry(); n != nil {
arrived[n.Name] = time.Now()
}
case <-deadline:
return false
}
}
}
createPeerEntry(t, filer1Address, "baseline")
require.True(t, waitArrival("baseline", 20*time.Second), "the baseline event never reached the subscriber")
// filer2 dies while no master is up, so no removal is ever broadcast
stopProcess(masterProcess)
stopProcess(filer2)
startProcess(t, weedBinary, filepath.Join(testDir, "master.log"), masterArgs...)
require.NoError(t, waitForLeader(masterPort, 60*time.Second))
require.NoError(t, waitForClusterNode(master, filer1Address, 60*time.Second))
diedAt := time.Now()
// Unpatched, every event from here on arrives 2 x LogFlushInterval late.
// The dead peer may hold reads for its unreachable bound plus the removal
// grace (2 x LogFlushInterval each); after that delivery must be prompt.
for i := 0; ; i++ {
name := fmt.Sprintf("probe-%d", i)
createPeerEntry(t, filer1Address, name)
if waitArrival(name, 10*time.Second) {
t.Logf("%s delivered promptly %v after the peer died", name, time.Since(diedAt).Round(time.Second))
return
}
if time.Since(diedAt) > 6*time.Minute {
t.Fatalf("aggregated delivery still held %v after the peer died unnoticed", time.Since(diedAt).Round(time.Second))
}
}
}
+15 -2
View File
@@ -2,13 +2,16 @@ package acl
import (
"context"
"net/http"
"strings"
"testing"
"time"
"github.com/aws/aws-sdk-go-v2/aws"
awsmiddleware "github.com/aws/aws-sdk-go-v2/aws/middleware"
"github.com/aws/aws-sdk-go-v2/service/s3"
"github.com/aws/aws-sdk-go-v2/service/s3/types"
smithyhttp "github.com/aws/smithy-go/transport/http"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
)
@@ -63,12 +66,22 @@ func TestBucketOwnershipControlsLifecycle(t *testing.T) {
putObjectOwnership(t, client, bucketName, types.ObjectOwnershipBucketOwnerEnforced)
assert.Equal(t, types.ObjectOwnershipBucketOwnerEnforced, getObjectOwnership(t, client, bucketName))
_, err := client.DeleteBucketOwnershipControls(context.TODO(), &s3.DeleteBucketOwnershipControlsInput{
deleted, err := client.DeleteBucketOwnershipControls(context.TODO(), &s3.DeleteBucketOwnershipControlsInput{
Bucket: aws.String(bucketName),
})
require.NoError(t, err)
raw, ok := awsmiddleware.GetRawResponse(deleted.ResultMetadata).(*smithyhttp.Response)
require.True(t, ok)
assert.Equal(t, http.StatusNoContent, raw.StatusCode)
assert.Equal(t, types.ObjectOwnershipBucketOwnerEnforced, getObjectOwnership(t, client, bucketName))
_, err = client.GetBucketOwnershipControls(context.TODO(), &s3.GetBucketOwnershipControlsInput{
Bucket: aws.String(bucketName),
})
require.Error(t, err)
assert.Contains(t, err.Error(), "OwnershipControlsNotFoundError")
putObjectOwnership(t, client, bucketName, types.ObjectOwnershipBucketOwnerPreferred)
assert.Equal(t, types.ObjectOwnershipBucketOwnerPreferred, getObjectOwnership(t, client, bucketName))
}
// A bucket with nothing stored already reports BucketOwnerEnforced, so this put has
+1
View File
@@ -0,0 +1 @@
.work
Loaded 100 of 475 files, more files were not shown because too many files have changed in this diff. Show more