A merged or closed PR kept every pull_request run going: concurrency groups
cancel a run only when a newer push arrives. After #12329 merged, 16 of its
runs were still queued or running. On pull_request_target closed, cancel the
queued and running pull_request runs on the PR's head branch whose commit
belongs to the PR.
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* s3tables: bind table bucket policy writes to the authorized bucket
PutTableBucketPolicy compared only the policy bytes, and
DeleteTableBucketPolicy removed the policy unconditionally. A bucket deleted
and created again under the same name between authorization and the write
took a policy change nobody authorized for it. Both writes now require the
bucket's CreatedAt and the policy they were authorized against, answer 409
otherwise and 404 for a bucket that is gone.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3tables: bind bucket tag writes to the authorized bucket
TagResource and UntagResource checked the creation time only for a table
ARN. On a bucket ARN the merged tags landed on a bucket deleted and created
again since authorization. extractResourceOwnerAndBucket now returns the
resource's CreatedAt for both kinds, and the write requires it.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3tables: bind bucket maintenance writes to the authorized bucket
PutTableBucketMaintenanceConfiguration wrote onto whatever bucket held the
name at write time. authorizeMaintenanceBucket now returns the bucket's
CreatedAt like authorizeMaintenanceTable, and putMaintenanceConfiguration
takes the check and creation time from both callers instead of an optional
closure.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3vectors: bind vector bucket policy writes to the authorized bucket
Put/DeleteVectorBucketPolicy authorized inside the compare-and-swap with
the owner from an earlier bucket read, so a bucket deleted and created
again by another account in between had its policy rewritten by the old
owner. The mutation now requires the entry's createdAt to match the
record it authorized against.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3vectors: bind DeleteVectorBucket to the authorized bucket
DeleteVectorBucket authorized one bucket record, then ran the trusted
catalog DeleteTableBucket on whatever bucket held the name by then, so an
empty bucket deleted and created again by another account in between was
deleted by the old owner. The catalog delete now accepts the authorized
bucket's createdAt through an in-process context value and treats any
other bucket as gone.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3vectors: bind CreateIndex and DeleteIndex to the authorized bucket
Both authorized the bucket, then ran trusted CreateNamespace, CreateTable
or DeleteTable on whatever bucket held the name by then, so an index could
be created in, or deleted from, a bucket another account created again in
between. Those catalog operations now honor the bucket binding
DeleteTableBucket already takes, and the vector handlers pass the
authorized bucket's createdAt.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* s3tables: bind table and view writes to the record they were authorized against
* s3tables: bind UpdateNamespace to the namespace it was authorized against
UpdateNamespace writes back the whole record it read, owner and creation
time included. A namespace deleted and created again before the write took
the first one's owner and CreatedAt, which also undoes the CreatedAt binding
creates now rely on. Refuse the write with 409 when the namespace at the
path has another CreatedAt.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
PUT and DELETE ?lifecycle were authorized as Write, so an identity granted
only object writes on a bucket could add expiration rules or remove the ones
the bucket administrator set. They now need Admin on the bucket, like the
encryption, public access block and ownership controls writes. Identities
authorized by IAM policies are unaffected: they are still checked against
s3:PutLifecycleConfiguration.
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
SigV4 verification picked the 32 MiB S3 Vectors body limit from the
credential scope, which is not verified until the body is hashed. A request
claiming the s3vectors scope on any other route could make the gateway buffer
32 MiB before authentication instead of 10 MiB. The vectors handler now sets
the limit on the request context, and every other route keeps 10 MiB.
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* s3api: hash an s3vectors body up to the Vectors API limit
SigV4 verification of a non-S3 request hashed at most 10 MiB of the body
and silently dropped the rest, so any S3 Vectors request over 10 MiB failed
its signature check with 403, although the API accepts 32 MiB. The limit
now follows the signing service, and a body over it is reported as
ErrEntityTooLarge instead of being truncated; the embedded IAM answers that
with InvalidInput, as it already does for a body over its own limit.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* s3vectors: answer an oversized request with a ValidationException
A body over 32 MiB was refused as "403 AccessDenied: authentication
failed". A declared Content-Length over the limit is now refused before
authentication, and a streamed body over it, whether the signature check
or the handler reads it, gets the same "request body exceeds 32 MiB"
ValidationException. The limit is the one SigV4 verification uses.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* s3vectors: log engine and catalog failures at the default level
An engine error reached the client as a bare "internal error", and the
cause was logged only with -v=1, so a misconfigured engine left nothing in
a default log. Engine and catalog failures are now logged as errors, and a
reported unavailability as a warning, with the operation, bucket and
index. The response still leaves the engine's message out, since it can
name internal hosts and paths, and points at the gateway log instead.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* s3api: hash the signed body through readRequestBody
streamHashRequestBody repeated readRequestBody's read-past-the-limit
check; it now reads the body with readRequestBody and hashes the bytes.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* s3api: encrypt the objects of a vector bucket declared AES256
CreateVectorBucket recorded encryptionConfiguration sseType AES256 and
returned it, but nothing on the write path read it: the index's data
files, manifests and transactions were stored without SSE.
A LANCE table bucket's record now supplies the bucket's default
encryption whenever the bucket has none of its own, so PutObject,
CreateMultipartUpload and CopyObject apply SSE-S3 to every object the
engine writes. The S3 bucket metadata itself is unchanged.
ICEBERG table buckets keep recording the declaration only: their
catalog reads the manifests clients upload straight from the filer,
which would see ciphertext.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* s3tables: refuse an AES256 vector bucket on a gateway without an SSE-S3 key
A vector bucket's AES256 is now applied to every index write, and SSE-S3
fails each write when the gateway has no KEK. Creating the bucket there
succeeded and the first CreateIndex came back as InternalFailure.
CreateTableBucket now refuses a LANCE bucket declaring AES256 when the
gateway reports no SSE-S3 key, naming s3.sse.kek and s3.sse.key;
CreateVectorBucket returns it as ValidationException. ICEBERG buckets,
whose declaration is only recorded, and tooling that sets no checker
are unaffected.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* s3api: refuse DeleteBucketEncryption on a bucket's declared encryption
GetBucketEncryption reports a vector bucket's declared AES256, but
DeleteBucketEncryption answered NoSuchBucketEncryptionConfiguration and
the declaration kept applying. The declaration is fixed when the bucket
is created, so the delete now returns MethodNotAllowed.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* test(s3vectors): keep an inherited WEED_S3_SSE_KEK out of the test cluster
The gateway refuses to start SSE-S3 with both s3.sse.kek and s3.sse.key
set, so a developer shell exporting WEED_S3_SSE_KEK made the AES256
bucket test fail.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* seaweed-vector: default --namespace to the gateway's Lance port
The old default, http://localhost:8888/lance, is a path nothing serves:
the Lance Namespace listens on its own port, 9101 by default on weed s3,
weed server, weed filer and weed mini. With it, the filer answered the
engine's catalog calls with a 500 and every CreateIndex failed with
InternalFailure.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* seaweed-vector: default --listen to a port no SeaweedFS server takes
127.0.0.1:8080 is the volume server's default port, so weed-vector next
to weed server exited with "Address already in use (os error 98)". 8090
is not among the defaults of master, volume, filer, S3 and its Iceberg
and Lance catalogs, IAM, WebDAV, SFTP, the message queue broker and
agent, the admin server or weed mini, nor any of their gRPC ports
(HTTP + 10000).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* lance: take any key while the S3 server runs without identities
Without identities the S3 port accepts any signature and treats the
caller as the admin. The Lance catalog still checked an x-api-key or
Bearer token against identities that did not exist, so weed-vector
started with --storage-option credentials on such a server got 401 on
every catalog call. The catalog now skips both checks when S3 auth is
off and leaves the caller to the S3 authenticator, which admits it as
the admin, as it already did for a request with no key. With
identities configured nothing changes: an unknown key is still 401.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* lance: tell clients to skip signing while S3 auth is off
A client with no credentials of its own could not reach a table's data
on a server without identities, though the S3 port takes unsigned
requests there: lance fell back to the AWS default credential chain and
failed with CredentialsNotLoaded. weed-vector without --storage-option
failed every CreateIndex that way. The catalog now adds
aws_skip_signature to the storage options it hands out while S3 auth is
off, and leaves it out once identities exist.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* seaweed-vector: bind the listener before registering with the master
The engine spawned its master registration and then bound --listen, so
an engine whose port was taken could reach the master before it exited:
on weed server with 18080 taken the master logged "vector node
127.0.0.1:18080 added" and removed it in the same millisecond, and the
gateway noticed it. Bind first, and name the address in the error
instead of "transport error".
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* seaweed-worker: keep an unsigned store off the AWS default chain
A store the catalog says takes unsigned requests needs no credentials,
but lance builds the AWS default credential chain for any store without
them, and the chain probes instance metadata on every dataset open.
With no credentials of its own, weed-vector spent a second on each
query (1030-1066 ms against 32-46 ms with --storage-option ones). The
merge of the operator's fallback and the vended options, which the
engine and the Lance worker each did in their own copy, moves into
table_options, and gives such a store empty credentials.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* test(s3vectors): serve an index with no identities and no engine credentials
A second cluster runs weed s3 without -config and weed-vector without
--storage-option, with no AWS credentials in either environment: weed s3
would turn them into an admin identity, and the engine would find them.
On master its CreateIndex fails with InternalFailure.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
A euclidean index answered QueryVectors with Lance's L2 distance, which is
the squared Euclidean distance: [3, 4] was 25 from the origin, not 5. AWS
defines the metric as the straight-line distance between the vectors. The
gateway now returns its square root, so the single engine and the enterprise
coordinator, which both hand back the raw Lance value, report the same
distance. Ordering is unchanged.
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* s3vectors: resolve the caller the way table buckets do
The vector checks used the identity name as principal, while the catalog
records a bucket's owner through getAccountID: the account id, or the
issuer-scoped user id of a federated identity. An identity with an account
or a federated identity did not own the vector bucket it created, a Deny
naming its account did not bind it, and an Admin identity in the admin
account did not get the admin bypass it has on table buckets.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* s3vectors: honor identity actions scoped to a vector bucket
An identity action such as s3vectors:QueryVectors:team-a granted nothing,
because the vector policy context never named the bucket. Table buckets
match <action>:<bucket> through the same context field, so pass the bucket
the check is about. Creating a bucket stays unscoped, as for table buckets.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* vector: build without a system protoc
lance's build scripts run $PROTOC and fail without one. Point it at the
protoc that protoc-bin-vendored ships, at the version in Cargo.lock, so
building inside seaweed-vector/ needs only the Rust toolchain. An explicit
PROTOC still wins, and a registry without the crate falls back to PATH.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* test(s3vectors): build weed-vector inside its workspace
Cargo reads seaweed-vector/.cargo/config.toml only when it runs there, so
build from that directory to pick up the vendored protoc.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* ci(vector): drop the PROTOC step the workspace config replaces
Builds run inside seaweed-vector/, where the cargo config already points
lance at the vendored protoc.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* test(s3vectors): resolve CARGO_TARGET_DIR before changing directory
A relative CARGO_TARGET_DIR given to make would point cargo, which now runs
in seaweed-vector/, and the cp after it at different directories.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* vector: pick the vendored protoc on every Linux host it ships for
protoc-bin-vendored also carries ppc64le, s390x and x86_32 binaries.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* shell: dial scrubbed volume servers on their registered gRPC port
volume.scrub, ec.scrub and volume.server.state built each data node's
address from its HTTP address alone, so a volume server whose gRPC port
is not the HTTP port + 10000 was dialed on the wrong port.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* shell: resolve -node volume servers to their registered gRPC port
An explicit -node/-nodes list was dialed on HTTP port + 10000 too.
Match each entry against the topology and take the data node's gRPC
port; an address the master does not know is used as given.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* shell: keep a gRPC port given in -node
A node given as host:port.grpcPort was rewritten to the registered gRPC
port; only a bare host:port takes it from the topology.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* shell: dial -node volume servers when the topology is unavailable
An explicit node list did not need the master before; keep using it as
given if the volume list cannot be fetched. Selecting every node still
requires the topology.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* shell: bound the topology lookup for an explicit -node list
With the master down, volumeServer.state -nodes waited on the master
forever instead of reaching the given volume servers. Give the lookup a
30s deadline and fall back to the addresses as given.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* shell: resolve volume server addresses from a topology-only listing
Only the data nodes are needed to find each server's gRPC port, so ask
the master for the topology without its volumes and ec shards. A large
cluster no longer risks running past the -node lookup deadline and
falling back to the HTTP port + 10000.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* shell: skip the topology lookup when every -node names its gRPC port
A host:port.grpcPort entry is dialed as given, so a list made only of
those does not need the master and no longer waits on it during an
outage.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* security: share the filer admin token check
CheckGrpcAdminPeer verifies an admin Bearer token inline. Move the check
into IsFilerAdminToken so an HTTP endpoint can accept the same token.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* lance: recognize the cluster's own components by a signed token
A request whose X-Seaweedfs-Cluster-Token header carries an admin token
signed with jwt.filer_signing.key comes from a component that holds the
cluster key: the vector engine or the Lance maintenance worker. Mark such
a request as trusted. It still authenticates as an identity, which stays
the principal for vended credentials. A dedicated header leaves the
Authorization header to Lance OAuth clients. A gateway with no key
trusts no one.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* lance: vend write credentials to a vector bucket's indexes only to trusted callers
An index in a vector bucket's vector namespace is also a Lance table, so
a caller granted s3tables:UpdateTable could describe it, or declare it
again, and get credentials that write the index data without passing the
vector bucket policy. In that namespace both now vend read-only
credentials unless the caller is trusted, and a declare from anyone else
no longer writes the dataset markers.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* lance worker: prove the cluster key to the Lance namespace
The gateway now vends write credentials to a vector bucket's indexes only
to callers that send a token signed with jwt.filer_signing.key. The
worker already takes that key for the admin stream; its namespace client
now mints a token from it for every catalog request.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* weed-vector: describe the index the gateway created and prove the cluster key
CreateIndex declared a table the gateway had just created, which only
worked because declare is idempotent. Describe it instead. The Lance
namespace vends write credentials to a vector bucket's indexes only to
callers that send a token signed with jwt.filer_signing.key, so take the
key with --filer-signing-key, the same flag the Lance worker has.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* lance worker: keep the cluster key out of Debug output
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* lance: answer a declare of an existing table with its own location
CreateTable succeeds for a name that already exists without looking at
the location requested, and declare kept using that location for the
dataset markers and the vended credentials. Declaring an existing table
with another table's location, such as a vector index, got credentials
to that table's data. Use the location the catalog holds, which
CreateTable now also reports for a new table.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* lance: sign cluster tokens with a key derived for the catalog
A cluster token was a filer admin token signed with jwt.filer_signing.key
and, by default, never expired. Sent to the Lance namespace on every
request, often over plain HTTP, a captured one would also authorize the
filer's IAM service. Sign it with a key derived from the cluster key for
this one use, which the admin services do not accept, and mint it with a
five minute lifetime. weed-vector drops its token lifetime flag.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* lance: follow a reloaded jwt.filer_signing.key
The Lance namespace copied the cluster key at startup. A SIGHUP reload
refreshes the gateway's filer guard but not that copy, so after a key
rotation the restarted vector engine and worker were no longer trusted.
Read the key from the guard on every check.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* iceberg: keep a vector bucket's indexes read-only
An index in a vector bucket is also a catalog table the Iceberg REST
catalog loads, and LoadTable vended credentials that could write it to
any caller granted s3tables:UpdateTable. A commit staged its metadata
file in the index directory on the same grant. Apply the vector
namespace rule to both, after the caller's own UpdateTable check.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* iceberg: confine a created table's location to its namespace before writing
Stage-create skips the registration that confines a location to the
table's namespace, yet vends write credentials for that location. A
caller allowed to create a table in any namespace of a bucket could name
another namespace's prefix, a vector index included, and get write
credentials to it. The non-staged create also wrote its first metadata
file there before the registration refused the location.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* lance: a declare that finds an existing table answers like describe
Creating a name that already exists is idempotent, so a caller allowed
only to create tables could declare someone else's table and get write
credentials to it, and clear its deregistered marker. Describe would have
vended that caller read-only credentials. A declare now looks the name up
first; when it only finds the table, writing needs UpdateTable, as it does
for describe.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* vector: compare json_array_contains elements as typed JSON values
Lance's json_array_contains compares each element's text with the value, or
with the value wrapped in quotes, so 1 matches "1", true matches "true", and
1.0 misses a stored 1. The filter swap now replaces it like json_extract and
json_exists: the value is parsed as JSON once per batch and compared with
jsonb's typed equality, and only literal arguments are swapped.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* s3vectors: match a filter value against the elements of an array
A filter never looked inside an array: {"tags": "space"} and
{"tags": {"$in": ["space"]}} both missed "tags": ["space", "classic"].
AWS documents that $eq, and so a bare value, matches an array metadata value
when any element equals the input. $eq, a bare value and each $in member now
also match through json_array_contains; $ne and $nin remain their negations.
Range operators, $exists and nested objects are unchanged.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* filer: scope a prefix JWT to the directory a zip extract writes under
Without a trailing slash, an extract request writes members beside the
named path, in its parent directory, while the JWT check only saw the
request path. A token scoped to /tenant1 could therefore extract into /
by posting to /tenant1?extract=zip. Add the extract root to the paths a
prefix-restricted token must cover.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* filer: do not keep an extracted archive under the name ".."
A multipart file name of ".." passes path.Base unchanged, so
extract.keep stored the archive as an entry literally named ".." in the
target directory, a name that local sinks resolve to the parent.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* leveldb3: keep a non-local bucket name out of the per-bucket folder
findDB splits the bucket out of a /buckets/ path and uses it as a folder
name under the store directory. The name cannot hold a slash and dot
names already go to the default DB, so this only rejects an empty or
non-local name, but it also lets static analysis see the folder cannot
escape the store directory.
* filer: clean the zip extract root before scoping and writing
A percent-encoded dot segment survives the path cleaning in front of the
mux, so a request path ending in "/" reached extractZipRoot with ".."
in it. The JWT scope check cleans the path it compares, but the members
were written under the raw root, so the two could disagree.
* leveldb3: apply the bucket folder check to bucket create and delete events
findDB already keeps dot and non-local bucket names out of the per-bucket
folders, but OnBucketCreation and OnBucketDeletion used the name as is. A
directory event for /buckets/.. opened a DB in the parent of the store
directory, and its delete removed that parent.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* leveldb3: keep a bucket delete event off the default DB
A bucket named _main maps to the default DB in findDB, so deleting
/buckets/_main closed the default DB and removed its folder, taking every
non-bucket entry with it.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* leveldb3: keep bucket names with a path separator out of the store dir
A bucket event named x/../_main passed the local-name check, and its
delete removed the default DB folder once store.dir/x existed.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* leveldb3: only reject the platform separator in a bucket name
A backslash is an ordinary file name character on Unix, and a bucket
folder named that way already exists for anyone who created one. Rejecting
it on every platform would route those entries to the default DB after an
upgrade. The slash can never appear in a bucket name here, and on Windows
the backslash is the separator, so rejecting os.PathSeparator alongside
the slash keeps the folder flat without changing Unix routing.
* leveldb3: check for an escaped bucket db before its delete event
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* s3 tier backend: bound connect and response-header waits
The tier client used the SDK default transport: a 30s dial timeout and no
limit on waiting for response headers. A blackholed endpoint cost about
two minutes per request with the SDK's retries, and an endpoint that
accepts connections but never answers held the request forever.
Dial with a 10s timeout and give up on a response whose headers do not
arrive within a minute.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* volume: stop the disk scan from waiting on a stalled tier backend
A volume server registers with the master only after every volume on its
disks has loaded, and loading a tiered volume reads its superblock from the
remote object. With the S3 endpoint unreachable, each read waited out the
client's timeouts and retries on one of the loader workers, so start-up grew
with the number of tiered volumes and the local volumes were not served
until it finished.
A tiered volume already loads without its superblock when the read fails.
Give that read 10s, and once it stalls, let the backend's other volumes skip
it for a minute instead of each waiting out the timeout. The volume reads
from the backend as before once the endpoint answers again.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* seaweed-volume: bound connect and read waits of the S3 tier client
Mirror the Go tier client: a 10s connect timeout instead of the SDK's
3.1s default, and a minute to receive a response, so an endpoint that
accepts connections but never answers cannot hold a read forever.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* seaweed-volume: stop volume loading from waiting on a stalled tier backend
Mirror the Go loader: the superblock read of a tiered volume gets 10s, and
once it stalls the backend's other volumes skip it for a minute. The
volume loads with its .vif version as before when the read fails, and
reads from the backend once the endpoint answers again.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* seaweed-volume: keep the .vif replication when a remote superblock read fails
Go applies the .vif replication to a tiered volume whether or not its
superblock read succeeded, so a volume loaded while its backend is down
still reports its replica placement. The Rust loader applied it only on
success and reported the default placement instead.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* volume: skip the .idx head repair while the tier backend is unreachable
The repair reads needle headers from the remote .dat, so after the
superblock read failed it would wait on the same backend again and hold
up the load. It runs on the next load that reaches the backend.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* seaweed-volume: skip the .idx head repair while the tier backend is unreachable
Mirrors the Go load: after the remote superblock read failed, the
repair's remote header reads would wait on the same backend again.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3 tier backend: give up a ranged read whose body stops arriving
The response-header timeout ends once headers arrive, so an endpoint
that answers and then sends no body held ReadAt, and a superblock
read's worker, forever. Cancel the request once the body makes no
progress for s3ReadTimeout.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* seaweed-volume: test that a tier read whose body stalls gives up
The SDK's stalled-stream protection already fails it, matching the Go
read's no-progress cancel.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* volume(rust): keep the S3 read timeout off upload_part
The SDK's read timeout runs from the start of the request, so a 64 MiB
part on a slow link would time out while its body is still being sent.
Go's ResponseHeaderTimeout only starts after the body is written.
upload_part now keeps the connect timeout and drops the read timeout;
stalled-stream protection still fails an upload that stops moving.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* volume(rust): bound the upload_part response wait from when the part is sent
Without a read timeout, an upload_part whose body was accepted but never
answered waited forever. The part body now goes to the connection in 64 KiB
chunks, and once the last chunk is taken the response must arrive within
S3_READ_TIMEOUT, like Go's ResponseHeaderTimeout. A retry rebuilds the body
and restarts that clock. The body keeps its in-memory contents, so signing
and the checksum header are unchanged.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* volume(rust): keep a retried part's response clock from an earlier attempt
The send state was a single flag, and a retry body could be rebuilt and
fully sent before the waiter saw the flag drop, so the waiter kept the
earlier attempt's deadline. Each body now carries its attempt number, only
the latest attempt marks the part sent, and the deadline holds only while
no later attempt has started.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* ci(release): publish weed-vector binaries
Tagged releases now carry weed-vector_linux_{amd64,arm64}.tar.gz, static
musl builds made the same way as weed-worker's.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* docker: carry weed-vector in the image
The image takes a pre-built weed-vector the way it takes weed-worker, or an
empty placeholder where none was built. `vector` runs it.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* ci(docker): build weed-vector for the images
The container workflows cross-compile weed-vector for amd64 and arm64
beside weed-worker and place it in the Docker context.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* install.sh: add the vector component
`--component vector` installs weed-vector from a release, and `all` adds it
when the release carries one, as it does for the worker.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* ci(docker): skip weed-vector for source refs that predate it
The manual latest-image rebuild accepts any source_ref; one without the
engine's server crate now builds its image without weed-vector instead of
failing the Rust job.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* docker: have the vector engine listen beyond loopback
weed-vector listens on 127.0.0.1 by default, which a gateway in another
container cannot reach. `vector` now adds --listen 0.0.0.0:8090 unless a
listen address is given by flag or WEED_VECTOR_LISTEN. The engine registers
its container address for a wildcard listen.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* localsink: skip keys that resolve outside the sink directory
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* filer: reject entry paths with dot segments
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* filer: test renames onto dot-segment targets
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* filer: keep the dot-segment rename test inside one bucket scope
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* localsink: resolve keys through an os.Root on the sink directory
The lexical check alone lets a symlinked directory that already exists
under the sink directory carry writes and deletes outside it. Create,
rename and delete now go through an os.Root opened on the sink
directory, so a symlink can only resolve inside it.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* localsink: follow absolute symlinks that resolve inside the sink directory
os.Root rejects every absolute symlink target, so a backup through an
absolute link that points back into the sink failed, and a delete through
it left the file behind. Resolve the parent of the key first and use the
resolved path when it stays under the sink directory. Operations still go
through os.Root, so a link swapped after the check cannot escape.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* operation: read a manifest chunk from the next replica when one fails
ChunkedFileReader read each chunk from one random location and returned
the first error, so a chunk whose cached replica went down failed the
whole file although another replica had it. Try the chunk's locations in
turn (local-data replicas first) and resume each attempt at the byte the
previous one reached. A failed write to the caller still ends the read.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* operation: drop a chunk volume's cached locations after a failed read
A chunk read that failed on a replica left that replica in the
10-minute location cache, so later reads kept trying it first. Drop the
volume's cached locations after a failure, and when every cached
replica has failed, ask the master once more and try the locations it
now lists.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* operation: don't continue a partly sent chunk on another replica
A volume server streams a large needle before it has verified the
checksum and aborts the response on a mismatch, so bytes that already
reached the reader may be corrupt. Continuing such a chunk on another
replica would hand back a mix of bad and good bytes as a complete read.
Move to the next replica only while nothing of the chunk has been
written; a mid-body failure still fails the read, and the dropped cache
entry lets the next read pick a live replica.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* operation: treat a short chunk body as a failed replica read
A replica that ends a chunk's body cleanly before the chunk's size made
the read look successful, so the next replica was never tried and the
following chunks landed at the wrong offsets. Read exactly the chunk's
remaining length and report a short body as an error.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* fix(s3): verify SigV4 signature with decoded path behind X-Forwarded-Prefix
When SeaweedFS is hosted behind a reverse proxy that strips a URL prefix (X-Forwarded-Prefix) and forwards the percent-encoded request path, signature verification fails for object keys containing non-ASCII characters. The client signs the canonical URI as the single percent-encoded path including the prefix, but the server verifies against the already-encoded forwarded path joined to the prefix, producing a mismatch (SignatureDoesNotMatch).
Decode the forwarded path and rejoin the prefix before verifying, mirroring the existing un-prefixed decoded retry. ASCII keys are unaffected because the decoded path equals the original, so the retry is skipped.
Signed-off-by: Mohd Quamar Tyagi <104281681+Tyagiquamar@users.noreply.github.com>
* s3: build the SigV4 candidate paths once and try them per host
The forwarded-prefix and plain branches each repeated the decode-and-verify
step for every host candidate. List the escaped and decoded paths once, the
prefixed forms first, and verify them in the same order as before.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Signed-off-by: Mohd Quamar Tyagi <104281681+Tyagiquamar@users.noreply.github.com>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* s3tables: let DeleteTableBucket remove empty leftover directories
Page through the bucket instead of reading one page, and check each bare
directory a deleted namespace left behind for files: empty ones go with the
bucket, one holding files refuses with 409 naming it.
The delete claims the bucket before scanning, and CreateNamespace refuses a
claimed bucket and confirms it again after its write, so a namespace created
meanwhile is seen by the scan or turned away instead of deleted with the bucket.
The bucket is removed only while it still carries that claim, otherwise 409.
A delete that refuses lifts the claim, retrying a failed release, since a claim
left behind refuses every create in the bucket.
The new namespace stays marked unfinished until that check passes, so no table
or view is created in one that gets backed out; a create takes over a namespace
still carrying the mark. Both checks accept only the bucket the create was
authorized against, by its creation time: one recreated under the same name
answers 404 and gets nothing written into it.
The removal is not recursive. Each leftover is claimed first, only while it is
still the directory the scan saw, and goes only while it holds no file or
catalog record; the bucket goes only while nothing is left in it. A file
uploaded or a namespace taken back after the scan is refused with 409 instead
of deleted.
* s3tables: keep directory-backed S3 objects in DeleteTableBucket
* s3tables: keep a file that replaced a leftover directory in DeleteTableBucket
* s3tables: claim only the authorized bucket in DeleteTableBucket
* s3tables: assert the prefix-object mark when claiming a leftover directory
* s3tables: hide unfinished namespaces from readers and DeleteNamespace
* s3tables: stop CreateNamespace on any failed bucket check
* s3tables: retry lifting claims on leftover directories
* s3vectors: declare the index namespace again after losing it to a concurrent declare
CreateNamespace now writes a namespace marked unfinished and confirms it
after checking the bucket. A second CreateIndex that declares the namespace
in that window takes the marked record over, and the first one's
confirmation answers 409, which failed its CreateIndex. Declare it again,
a bounded number of times, when the create answers a conflict.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3tables: name an unfinished namespace that keeps a bucket from being deleted
A CreateNamespace that dies between its write and its confirmation leaves
a namespace marked unfinished. Listings and GetNamespace hide it and
DeleteNamespace answers that it does not exist, while DeleteTableBucket
counts it, so the bucket could not be deleted and nothing said why. Name
it in the BucketNotEmpty answer; creating it again takes it over.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3tables: read a refused empty-directory delete with filer.DeleteEntryError
Use the filer's own reading of a non-recursive delete's answer instead of
a copy of its message prefix.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3vectors: release the held create when the concurrent-declare test fails
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3vectors: buffer the held create's result so it can exit after a failure
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3tables: retry lifting a refused namespace delete's claim
A claim left on a namespace refuses every create in it, the same harm the
bucket's claim does, so its release retries too. With no one-shot caller
left, the retrying release takes the releaseDeleteClaim name.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3tables: keep the data under a hidden directory on bucket delete
The scan filed every hidden entry by name for a recursive delete, so .hidden/data went with the bucket. A hidden directory waits with the leftovers instead: the walk refuses one that holds a file or an object, and removeEmptyTree takes only one that is still bare and empty. A hidden file is removed by name as before.
---------
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Chris Lu <chris.lu@gmail.com>
* lance worker: let dataset::open take an object store wrapper
open_with is open with the dataset's object store wrapped, so a caller
such as weed-vector can put a read cache in front of the table's files.
open keeps its signature and opens unwrapped, as before.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* vector: cache reads of write-once Lance files across requests
Every VectorService call opens the table afresh, and Lance keeps no
data pages between opens, so each QueryVectors re-read the whole
vector column through the S3 gateway: 83 MB and about 228 range GETs
per query at 50k x 384. The gateway and volume server spent their CPU
streaming it and throughput stayed at 6-8 queries/s whatever the
client concurrency.
weed-vector now wraps each table's object store with a bounded
in-memory cache (--read-cache-mb, 512 MiB by default, 0 disables). It
only serves data files, deletion files and index files: Lance writes
each once under a fresh random name, so a cached read cannot go stale,
even when an index is dropped and recreated at the same location.
Manifests, tags and every other path still go to the store, and each
call still describes the table, loads the latest version and takes
the query lease, so reads see every write committed before them.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* vector: cache an owned copy of each fetched read
A range from get_ranges can be a slice of a larger response buffer,
such as a coalesced fetch, and caching the slice kept the whole buffer
alive while the weigher counted only the slice. The cache now copies
each fetched range and whole-object read into its own buffer, so
--read-cache-mb bounds what it actually holds.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
The volume server views summed the logical size of the volumes into
DiskUsage while DiskCapacity came from the filesystem statfs size. On a
compressing filesystem the logical sum exceeds the physical capacity, so
the server details popup derived a negative available space and rendered
it as "NaN undefined".
Pair each disk's capacity with the usage measured the same way, using the
statfs numbers the volume server already heartbeats, as CollectTierStats
does. Volume servers without statfs numbers keep the previous slot-based
capacity and logical bytes, and EC shard bytes are only counted there
instead of on top of the filesystem usage.
* test(s3vectors): add an opt-in end-to-end perf suite
perf_test.py runs on the same cluster fixture as the integration suite
(Go or Rust volume, single or distributed queries) and times the data
plane through the S3 Vectors API: PutVectors ingest and upsert, QueryVectors
latency with and without metadata and a 10% filter, query throughput
across client threads, GetVectors, segmented ListVectors, and
DeleteVectors. Recall is checked against an exact client-side search and
listings must be complete; timings only report. Sizes come from PERF_*
variables, and results go to a JSON report and a printed summary.
It runs through run_perf.sh or `make perf`. run_test.sh does not collect
it, so CI is unchanged.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* test(s3vectors): check every listed key in the perf suite, not just the count
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* test(s3vectors): cap perf samples at PERF_VECTORS so small runs work
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* test(s3vectors): warm perf worker clients before starting throughput timers
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* test(s3vectors): report the key count of perf GetVectors and DeleteVectors calls
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* vector: parse a filter's JSONPath once per batch instead of per row
Lance's json_extract and json_exists parse their JSONPath argument again
for every row they see. Every translated S3 Vectors filter goes through
them, so a flat scan paid about 1 µs per row before the predicate
compared anything, and a 10%-selective {"category": {"$eq": "c3"}} ran
about 50% slower than the same query unfiltered.
The path is always a literal, so the scanner now parses the filter as
before and swaps in copies of the two functions that parse it once per
batch. They call the same jsonb selector and serializer, so results and
errors do not change. Calls whose path is not a literal keep Lance's
functions. The filter SQL is unchanged, so the gateway, existing tables
and the distributed workers (which run query::ann) need no change.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* s3vectors: read the field once per extractor in $in and $nin
$in and $nin rendered one IN per member, each with its own json_extract,
so an N-member list parsed the row's metadata N times. Members that share
an extractor (strings and booleans, or numbers) now go into one IN list,
so a list reads the field at most twice. Matching is unchanged: each
group still only sees its own JSON type and still coalesces NULL before
$nin negates it.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* s3vectors: rename the $in literal map so codespell passes
Codespell reads "lits" as a misspelling of "list".
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* volume: serve a ranged GET without reading the whole needle
A ranged GET read every needle up to 1MB in full, checksum and all,
then copied out the requested bytes. Read the metadata and stream only
the range instead, as the Go server already does above 1MB and the Rust
server does at any size. A small range of a 790KB needle goes from
~340us to ~160us.
The metadata read stays non-terminal: the range read that follows
reports the io outcome, so a failing disk still builds its error streak.
HEAD, compressed needles and chunk manifests keep their current reads.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* volume: read the whole needle for a range that may need an image transform
Without a URL extension, the stored name can still make a ranged GET
with width, height or crop parameters an image transform, which needs
the payload. Keep the full read there instead of streaming the range.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* volume: stream a range from the record its metadata came from
The streamed read looked the needle up again, so a replacement written
after the metadata read sent new bytes under the old length, checksum
and headers. Read the record the metadata read resolved, as the Rust
server does, and after a compaction refuse a record whose size changed.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* rust volume: read the whole needle for a range that may need an image transform
A ranged GET with no URL extension but width, height or crop parameters
took the metadata-first path, so a stored image name never got its
transform. Read a small needle whole there, as the Go server does.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* volume: keep later parts of a multi-range reply on a moved record
When a compaction moved the record before one part of a multi-range
reply, that part re-resolved it, but the next part went back to the
offset the metadata read saw. Remember the re-resolved offset too.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* volume: check a compaction-moved record is the one the metadata came from
After a compaction, a range re-resolved the needle and accepted any record
of the same size, so a replacement written in between could send its
bytes under the old headers. Compare the moved record's checksum and
append time with the metadata already read, and fail otherwise.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* volume: accept a re-encoded checksum and count read errors on a moved record
Compaction rewrites a pre-3.09 checksum in the current encoding, so the
moved-record check now accepts either. A failed read of the moved
record's metadata now feeds the io-error streak and surfaces as that
error instead of not-found. The pinned record is a fresh value rather
than a write through the index's pointer.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* volume: keep a moved record's wrapped offset when a range follows it
With 4-byte offsets, a record past the 32GB wrap is read at offset plus
MaxPossibleVolumeSize. The compaction refresh dropped that, so it checked
and streamed the wrong location. Retry the wrapped offset in the moved
record check, as the metadata read does, and apply it to the stream.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* vector: scan only the fragments a ListVectors page covers
A page pushed its offset into the scan, but the scan still loaded every
fragment first, reading each one's deletion file: about 105 S3 GETs per
page at 50k rows against 25 at 10k. The page's fragments now come from the
live row counts in the manifest, so a page reads only its own. Tokens and
row order are unchanged; a manifest without the counts keeps the old scan.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* s3vectors: list a ListVectors segment in the engine
The gateway paged through the whole index for every segment and kept one
row in segmentCount, so a segment cost segmentCount engine pages and 16
segments listed slower than one. The engine now takes segment_count and
segment_index: it reads keys across the stride, keeps the segment's rows
and takes their data and metadata alone, one call per page.
A segment is still the rows whose position modulo segmentCount is
segmentIndex, and its token still the position to resume from. An engine
that predates the fields does not set segmented, and the gateway then
pages through it as before.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* vector: read only the key column across the table in GetVectors
The key lookup scanned every column. On S3, lance reads a column during
the scan when a row of it is under 1000 bytes, so a 128-dimension vector
was read for every row of the table to return 100 of them, and data and
metadata were read even when the request did not ask for them. The scan
now projects the requested columns and materializes everything but the
key late, for the matching rows only.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* s3vectors: start the legacy segment fallback from the page already read
With an engine that ignores segment_count, the gateway dropped the page
it had just read and fetched it again. The fallback now consumes that
page first, so a one-row segment page still costs one engine call.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* vector: check metadata on segmented ListVectors pages
The segment test listed data only. Its rows now carry metadata on two in
three keys, and every segmented row, read through take_rows, must equal
the same row in a plain listing.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* vector: lease the version while a segment takes its data
A segmented page with data or metadata reads keys first and then takes
the other columns from the same version, so maintenance could reclaim
its files in between. That path now holds a lease like GetVectors. A
keys-only page is one scan and stays unleased, as plain pages are.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* vector: leave GetVectors materialization to lance
Forcing every column late made a 100-key GetVectors with data slower at
10k x 128 in the perf cluster: 54.7 ms against 46.3 ms. The take issued
a read per row, where lance's heuristic reads a narrow vector column in
one pass per fragment. The projection keeps the gain on its own: with
neither data nor metadata requested, the call went from 38.6 ms to
25.3 ms and from 150 to 89 S3 GETs. At 50k x 384 the vector is already
late under the heuristic, and a full lookup is unchanged.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* vector: follow a leader hint at once and skip one that failed this pass
A follower naming a dead leader stalled master failover. Each pass broke
off at the first hint, tried that leader first on the next pass, then
got the same hint from the same follower, so the later configured
masters were never tried. A pass now works like wdclient.tryAllMasters:
it follows a hint at once, does not follow a hint to a master that
already failed in that pass, and moves on to the rest. The hops are
bounded by the master count, as the volume heartbeat bounds its hints,
so two masters naming each other cannot spin.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* vector: test a hint naming a failed master in its gRPC form
A pass keys failed masters by host:http, so a configured m1:9333 that
failed must also stop a hint of m1:9333.19333. Cover that equivalence.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* vector: refuse to drop a location that is not the table's own
DropIndex removed everything under whatever location the catalog
described. That location is caller-supplied through the Lance
namespace and only confined to the bucket, so a table registered at
s3://bkt/vector, or at another index's directory, wiped every index in
the bucket or that index. drop now accepts only the colocated path the
gateway's tableLocation derives from the table id, after trimming a
trailing slash, and the RPC answers FailedPrecondition otherwise. This
mirrors the ancestor and live-entry guards on the catalog's DeleteTable.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* vector: bound the master dial and try a leader hint once
A remembered leader hint was never cleared and the dial had no connect
timeout, so a black-holed former leader added the kernel's SYN timeout
(about two minutes) to every reconnect pass before the configured
masters were tried, and registration lapsed. The dial now gives up
after 5s, the same bound wdclient puts on a master call, and a hint is
tried once ahead of the configured masters, as wdclient.tryAllMasters
keeps nextHintedLeader for one cycle only.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* vector: refuse a master port whose gRPC port would pass 65535
grpc_addr derived the gRPC port as http + 10000 in u16, which panics in
debug builds and in release silently dials a wrapped port for HTTP
ports above 55535. Go's pb.ServerToGrpcAddress adds in int, so the port
never wraps and the dial fails instead. The addition is now checked,
and weed-vector refuses to start with an error asking for the explicit
host:http.grpc form.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* vector: refuse a numeric master port past 65535
A master given as m1:65536 or m1:9333.65536 failed both u16 parses and
fell through to the raw address, so startup accepted it and every dial
failed. Ports now parse as u64, as Go's ServerToGrpcAddress does with
ParseUint, and one outside u16 is refused at startup.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* vector: bound the master's first keep-connected answer
The connect timeout only covers TCP. A master that accepts the stream
and never answers held stream_to forever: grpc-go sends the response
headers with the first message, so the call never returned and later
seeds were never tried. A healthy master always answers at once (a
follower with its leader hint), so the stream open and the first
response now share the same 5s bound; after that the long-lived stream
stays untimed.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
s3tables: keep the fence on a declare of an existing index
Deciding the existing-index exception before the handler ran left a
window: an index deleted between that lookup and the handler's own check
would be recreated, unfenced, by a caller without s3vectors:CreateIndex.
Drop the up-front exception, so every CreateTable runs fenced and a
fresh create is refused at its first write. The engine's declare still
passes because the idempotent path writes nothing; for a build whose
idempotent path re-applies state already in place, the fence answers an
UpdateEntry that leaves the entry exactly as stored without writing.
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* s3tables: make the vector namespace of a vector bucket read-only
S3 Vectors authorizes index operations against the vector bucket policy,
but an index is also a catalog table in the bucket's "vector" namespace,
and the S3 Tables API only checks s3tables grants. A caller the vector
policy denies DeleteIndex could still delete the index, and its data,
with DeleteTable.
Untrusted writes that name the vector namespace of a LANCE bucket now run
behind a fence that passes filer reads and refuses filer writes. The
operation's own checks still answer everything short of a write, so a
caller without the grant gets its usual denial and learns nothing about
the bucket, and a CreateTable that finds the index already there, which
is what the vector engine's declare does, still succeeds. A refused write
answers 403 AccessDenied. Trusted callers (the gateway, the shell) are
unaffected, and so is a namespace named vector in an Iceberg bucket.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* lance: refuse register and deregister in a vector bucket's vector namespace
Every catalog write the Lance namespace makes goes through the S3 Tables
handler and its vector namespace fence, but register over an existing
table at the same location and deregister only add or remove a marker
file in the dataset directory. Check the vector namespace before those
writes, after the table read that already gates them.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3tables: refuse deferred creates in a vector bucket's vector namespace
Manager.AuthorizeCreateTable gates the Iceberg create paths that write
metadata into the table bucket before the catalog registers the table.
Those writes never pass the handler's fence, so the gate applies the same
read-only rule once the caller's own create permission is confirmed.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3tables: let a declare of an existing index past the vector namespace fence
The vector engine declares every index the gateway has just created, and
that CreateTable only reaches its idempotent path. Upstream that path
writes nothing, so the fence never trips, but a build whose idempotent
path repairs table state would refuse the engine. Decide the exception
up front instead: a CreateTable naming a finished index in the vector
namespace runs unfenced, while a new name or an unfinished create stays
fenced.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* vector: throttle worker discovery from lookup end
cli() set nextResolve before resolve() ran, so a lookup that takes as
long as the throttle interval (an unreachable first master costs the
full 5s dial timeout) left the next lookup already due when it
returned. Discovery then ran back to back and, since it holds d.mu,
every vector request queued behind it.
resolve() now sets nextResolve when the lookup lands, using the short
recheck window only while no client is cached.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* vector: resolve workers outside the discovery lock
resolve() held d.mu across the master lookup, so while it waited out an
unreachable master every vector request blocked on cli(), even with a
healthy worker already cached.
The lookup now runs in a goroutine without d.mu, and d.resolving keeps
it to one at a time. Callers with a cached client use it while the
lookup is in flight; callers without one wait for that lookup and share
its result. Throttling, the sticky worker and reset(c) are unchanged.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* vector: keep the cached worker connection when a redial fails
resolve() closed d.conn before dialing the replacement worker. A
synchronous dial error then left d.client bound to a closed connection.
Requests got codes.Canceled, which never triggers reset, and later
refreshes saw the unchanged d.worker and never redialed. The old
connection is now closed only after the new dial succeeds.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* s3vectors: bound ListVectors maxResults and segment parameters
segmentCount had no upper bound, so segmentCount=1000000,
segmentIndex=999999 made the segment emulation scan the whole index to
return nothing. Apply the AWS ListVectors windows before authorization
and any engine call: maxResults 1..1000, segmentCount 1..16,
segmentIndex in [0, segmentCount), and segmentCount and segmentIndex
given together. The fields become pointers so an explicit 0 is rejected
while an omitted one still takes the default.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3vectors: store and return distanceMetric in lowercase
CreateIndex stored distanceMetric exactly as sent, so an index created
with "COSINE" came back as "COSINE" from GetIndex and QueryVectors.
AWS returns the lowercase enum values. Lowercase the metric when the
index is created, and again when the record is read so indexes already
stored in mixed case render the same.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* s3vectors: authorize CreateVectorBucket as an s3vectors action
CreateVectorBucket went through the catalog's CreateTableBucket check, so
an identity granted s3vectors:* was denied the create while s3tables:*
was allowed it. Check s3vectors:CreateVectorBucket in the gateway, the
same way the catalog checks a bucket create, and create the record on
the internal manager under the caller's name so the caller stays the
owner.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3vectors: delete only the index record DeleteIndex checked
DeleteIndex read the record, checked it is a Lance index, then deleted
by name. A table that replaced the index in between was deleted too.
Send the checked record's version token, so a replaced record fails the
delete with a 409 ConflictException.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3tables: reject s3vectors rules in any letter case
validatePolicyDocument kept s3vectors actions and resources out of the
shared policy slot with case-sensitive checks, so "S3Vectors:GetIndex"
was stored as a rule that never takes effect. IAM service prefixes are
case-insensitive; fold case in both checks.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* ec.balance: spread concurrent shard moves over target nodes
The planner gives every freshly encoded volume the same target order, so
running each volume's moves in plan order sends all concurrent copies to the
same node while the other nodes idle. Pick, among the volumes not already
moving, the pending move whose target receives the fewest moves right now.
One volume still never has two moves at once, and moves of one shard keep
their plan order.
* ec.balance: let a move into a node wait for the move of its volume leaving it
The planner counts the slot an outgoing shard frees as room for a later
incoming shard of the same volume, and the receiver refuses a shard it has no
slot for. Make the spread tests deterministic: hold the first moves until all
have started, and check that errors of several volumes are all returned.
* ec.balance: pin that the spread moves run concurrently and in plan order without parallelism
* wdclient: publish applied volume-location deletes to registered listeners
The master already tells each connected client which volumes a server
stopped hosting; only the MasterClient's own vidMap applied it. Expose
the same filtered deletions (moves between a server's disks excluded)
through OnVolumeLocationDelete so clients keeping a second location
cache can drop the retired server too.
* wdclient: drop retired volume servers from the FilerClient location cache
The S3 gateway's FilerClient keeps its own vidMap, filled by filer
lookups, and only forgot a retired volume server after a read against
it failed (a ~15s header-timeout stall when the host still resolves).
When built with a MasterClient, subscribe to its applied deletions so
the cache forgets the server as soon as the master reports it gone.
* wdclient: drop retired EC servers from the regular location map too
Filer lookups cache EC locations alongside regular ones in vid2Locations,
so an EC removal has to clear the retired server there as well as in the
EC map for the delete to reach what reads actually use.
* wdclient: tombstone pushed deletions against in-flight provider fills
A filer lookup already running when a delete lands can return the retired
location afterwards and re-cache it. Keep a short tombstone on each
deleted (vid, url) pair so fills neither cache nor return it while the
lookup source is still settling.
* wdclient: clear the FilerClient location cache on master reconnect
Removals the master pushed while the gateway link was down never arrive
as deletes, so on reconnect the lookup-fed cache is dropped outright
rather than kept generation-wise like the stream-fed one, and relearned
through filer lookups as before.
* wdclient: invoke push hooks outside their registration locks
A callback that registers or replaces a hook would wait on the read lock
the invocation already holds and stall the master stream. Copy the hook
under the lock and call it after the lock is released.
* wdclient: report pushed adds so confirmed re-adds clear the suppression
A server coming back at the same URL inside the tombstone window stayed
hidden from provider fills. updateVidMap now carries the additions it
applied through the same hook (renamed OnVolumeLocationChange), and the
registered cache untombstones them without eagerly caching them.
* wdclient: amortize tombstone expiry over a delete batch
A server leaving takes every volume it hosted with it, and scanning all
recentDeletes per removed vid squared the batch. Prune lazily on reads
and at most once per half TTL on writes.
* wdclient: let every location cache subscribe to pushed volume changes
The setters were single-slot, so a second FilerClient built on the same
MasterClient would have silently replaced the first one's subscription.
Registration now returns an unhook func, delivered to all listeners, and
FilerClient.Close releases its own.
* wdclient: hold the deletion fence until the master re-confirms a location
* wdclient: check the deletion fence and the cache add under one lock
* wdclient: drop provider answers that crossed a cache reset
* wdclient: bound the deletion fence by evicting its oldest mark
* wdclient: refresh a repeated removal instead of evicting another mark
* wdclient: evict the oldest deletion-fence mark in constant time
* wdclient: fence a server only once it serves the volume in no form
ec.encode registers the shards before the regular copy is deleted from the
same server, and ec.decode mounts the regular copy before the shards come
off it. Both arrive add-first in separate heartbeats, so the later removal
tombstoned a (vid, url) pair the server still serves and dropped it from
the FilerClient map; the filer's answer naming that server was then fenced
out and the volume read as not found until the gateway reconnected.
Tombstone and report a removal only when neither map still names the url
for that volume.
* wdclient: serve a provider answer that crossed a cache reset
The answer is as fresh as the provider's view; only the cache it would
land in has been re-told. Dropping it failed every read whose lookup was
in flight during a master reconnect with 'volume not found'. Return it
and leave the new view to be filled by the next lookup.
* wdclient: test the moved volume whose old server keeps proxying
* wdclient: stop caching inside the master lookup provider
The provider wrote every answered location through addLocation, which
clears the deletion fence: a lookup still naming a server a pushed delete
had just retired untombstoned and re-cached it. Fallback lookups already
gate caching through fillLocation, and authoritative lookups bypass the
cache by contract, so the provider no longer writes the map at all.
* wdclient: count only master-pushed locations when suppressing a removal
hasLocation could not tell a confirmed regular copy from a lookup fill:
the master answers an EC volume with its shard servers, so the regular map
holds shard locations that arrived by lookup, and entries an earlier
generation learned linger past a reconnect. An EC removal whose url still
showed either was suppressed, leaving the retired server served.
The vid map now records the generation each regular location was pushed
at, drops an unconfirmed regular entry alongside the shard that fed it,
and answers hasServingLocation from pushed marks and current-generation
entries only.
* wdclient: test removals against lookup-cached and stale locations
* wdclient: fence removals only in the gateway cache
The fence and the pushed-location marks ran inside every MasterClient,
including the filer's, whose map is also filled by lookups and cleared
by InvalidateCache. A removal reported while the server still served the
volume (e.g. a decoded copy whose mark a failed read had wiped) left the
pair fenced until the master pushed it again, with reads failing as
volume not found meanwhile.
Only the S3 gateway's FilerClient now tombstones, from the removals its
push-only MasterClient reports. That client decides a removal from the
current generation of both its maps after the whole message is applied,
so pushedLocations goes away, and the master lookup provider caches as
before. The hooks follow the existing single-slot SetOn...Fn pattern and
are wired next to SetOnPeerUpdateFn.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* wdclient: let a removal fence expire after a minute
The master drops a broadcast when a client's channel is full, so the
re-add that would lift a fence can be lost while the gateway stays
connected. A mark held until that re-add then hid a live server, and a
volume whose only copy it held read as not found until the next master
reconnect. Bounding marks by a minute still outlasts a lookup in flight
when the removal landed; expiring early only falls back to learning by a
failed read, as before this change.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* s3 sink: forward user metadata to the destination object
filer.backup and filer.sync to an S3 sink copied the object body and tags
but dropped the X-Amz-Meta- attributes the S3 gateway stores with each
object, so the destination object came back without its user metadata.
Send them as the upload's metadata, keyed by the lowercased suffix.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3 sink: send mtime as metadata without writing it into the entry
The sink stored X-Amz-Meta-Mtime into the replicated entry's Extended map,
modifying the caller's entry, and since tags were limited to X-Amz-Tagging-
keys it was never sent anywhere. Add it to the upload metadata instead,
keeping an mtime the source object already carries.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3 sink: RFC 2047 encode metadata values that are not printable ASCII
The S3 gateway keeps user metadata values as raw bytes. Forwarded as is, a
UTF-8 value fails SigV4 on AWS and a control character makes the HTTP client
reject the request, and filer.backup retries such an object forever. Encode
those values as RFC 2047 encoded words, which S3 decodes on write.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3 sink: upload without user metadata past the S3 2 KB limit
The S3 gateway does not cap user metadata, but S3 rejects more than 2 KB of
it, and filer.backup would retry that object forever. Upload such an object
without its metadata and log a warning, as the sink did before metadata was
forwarded.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3 sink: skip metadata keys that are not valid header names
Filer gRPC clients can store any Extended key. An X-Amz-Meta- key with a
space or control character makes the HTTP client reject the whole upload,
which filer.backup retries forever. Leave such keys out with a warning.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3 sink: pick the same value for metadata keys that differ only in case
An entry holding both X-Amz-Meta-Project and the legacy x-amz-meta-project
maps both to project, and map order decided which value was uploaded. Keep
the lowest key, so the canonical form wins over the legacy lowercase one.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3 sink: use a test value codespell accepts
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* master: undo the stored record when a delete report arrives
Deleting a volume decremented the active count from the delete report's
ReadOnly flag, which the volume server never sets, so a volume the master
had already stopped counting as writable was uncounted a second time.
Every volume.move does this to its source, and the node, rack, DC and
topology counts drifted negative; the full-list path already used the
stored record. Undo exactly what was counted for the record held.
* volume: say whether a deleted volume was read-only
Explicit deletes, unmounts, collection deletes and the departures named
in a delta heartbeat all left ReadOnly unset, so an older master undid
the writable count for a volume it was no longer counting. Carry the
flags, as the Rust server already does.
* ci: share the musl Rust target setup as a composite action
Three container workflows carried the same toolchain, musl-tools and
aws-lc-sys link shim block verbatim.
* release: ship weed-worker as a static musl binary
The gnu build inherits the runner's glibc, 2.39 since the move to
ubuntu-24.04, so the released worker failed to start on Ubuntu 22.04
with 'GLIBC_2.38 not found'. The container images already build it
against musl with the same recipe.
* release: ship the Linux weed-volume as a static musl binary
Same glibc trap as the worker. The arm64 job also installed libssl,
which nothing in the volume links.
* ci: resolve the manual latest build's actions from master
Its checkout is an arbitrary source ref that may predate a local
action; the workflow itself always runs from master.
* filer: resume a rejoining peer's meta subscription at its last position
OnPeerUpdate starts every peer add at the update's arrival time. For
filers sharing one store there is no persisted offset to correct it, so a
peer that drops out and rejoins (a graceful restart) resubscribes from
"now" and the events it logged while its stream was down never reach the
aggregated buffer - filer.backup and other meta subscribers silently
skip them.
Track each peer's delivered-through position (the same timestamp domain
as the stream's events and idle-heartbeat claims) and rewind the start
of a rejoining subscription to it. First-seen peers keep the given
start, and non-shared peers still resume from their persisted offset.
* filer: keep the re-add resume point behind a replay-floored event
The session's replay floor already pins the persisted offset and the
reconnect resume point so an event that outlived its retry budget is
redelivered on resubscribe; cap peerResumeTsNs to the same position on
session exit so the new re-add resume cannot skip past it either.
* filer: stop test aggregators' log-buffer workers
* filer: cover peer rejoin resubscribing at the resume position
* filer: drop the unreachable replay-floor cap on the resume point
Only a peer with its own store can floor an event, and that peer's
subscribe start always comes from its persisted offset, never from the
re-add resume point. The cap also ran in the old session's defer, after
the re-add had already read the value.
* filer: say why a re-add may resume from an idle heartbeat
The heartbeat comments promised a reconnect never resumes from one,
while the re-add resume point does. Spell out the difference and the
fence that makes the local-stream claim safe to resume from.
* filer: forget a removed peer's resume point after an hour
The map held an entry for every address ever seen; with pod IPs that
is one per restart. The removal mark now outlives the watermark grace
and takes the resume point with it once the retention passes.
* filer: advance the resume point only for tracked peers
Same rule as the watermarks: a straggler from a dropped session must
not recreate the entry.
* filer: set the test aggregator's subscribe loop instead of a nil fallback
* fix(s3tables): keep the data under a deleted namespace
A rename leaves the source table directory holding data the renamed table
still points at, and DeleteNamespace removed it recursively. The delete now
keeps the directory, minus its catalog record, when files remain under it;
CreateNamespace can take it back.
* fix(s3tables): keep hidden-directory data, report name takeover as 409
- DeleteNamespace also looks for files under hidden directories, since a
table location may sit under one.
- CreateNamespace answers 409 when a file takes the name during the
take-back of a kept directory.
* s3tables: a vanished leftover holds nothing to keep
A bare directory removed between the namespace listing and the walk
made dirHoldsFiles answer its ListEntries NotFound, and the delete
reported that miss as a 404 or 500 on a live namespace. Skip the gone
directory instead: whatever it held is already gone, so it cannot keep
the namespace.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* s3tables: a delete's gone entry is a conflict, not a success
The filer answers a precondition delete on an entry that is already gone with FailedPrecondition carrying the not-found text, which DoRemoveRequest reads as a completed delete -- DeleteNamespace answered 200 for a delete that never ran. deleteDirectoryIfMatches now calls DeleteEntry directly, keeping the FailedPrecondition-to-conflict mapping and refusing a response-level error. keepAsLeftover likewise reports a directory removed since the scan as ErrConcurrentUpdate, so the handler answers 409 rather than 500.
---------
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: Chris Lu <chris.lu@gmail.com>
* s3: allow POST Object uploads when auth is off
With no identities configured every other handler skips authentication,
but PostPolicyBucketHandler still verified the form signature, so a
browser form upload failed with InvalidAccessKeyId (or a malformed
credential error without signature fields). Skip the signature check
when auth is off; the policy conditions are still enforced.
* s3: update AuthPostPolicy comment for the auth-off path
---------
Co-authored-by: Chris Lu <chris.lu@gmail.com>
* s3: take ListObjects marker/start-after as raw keys
Under encoding-type=url AWS url-encodes the response fields but treats
request markers as raw keys: SDKs decode Marker/NextMarker/StartAfter and
send the decoded key back. Unescaping the marker server-side read a literal
"%XX" key as the key it encodes — StartAfter=a%2Bb resumed after "a+b",
skipping it, and a paginator whose marker decoded to an earlier key got the
same NextMarker twice and failed.
Take the request value as-is; only the server's own base64: continuation
token still decodes. The marker-echo dedup now compares contents in the
response wire encoding and NextMarker in its raw form, since a raw "%XX"
marker equals the escaped emission of a different key.
* s3: take ListMultipartUploads key-marker as a raw key
Same raw-marker rule as ListObjects: under encoding-type=url the emitted
key markers are url-escaped keys, and a client decodes them before sending
the marker back, so the request value is already the raw key. Keeping the
token decode only for the non-url path, where the response could not carry
the raw key and falls back to the base64: token form.
* s3: cover ListMultipartUploads url key-marker at the handler
The selectMultipartUploads tests bypass the request marker decoding, so a
regression that unescapes an encoding-type=url key-marker would pass them.
Drive a listing through ListMultipartUploadsHandler with a literal
"a%2Bb" marker and assert it resumes after that exact key.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
A rule without MaxAgeSeconds was stored as 0, so GetBucketCors returned
MaxAgeSeconds 0 and preflights sent Access-Control-Max-Age: 0. Store -1
for unset, which corsRuleFromProto already reads back as nil.
getObjectResources and getBucketMultipartResources dropped the strconv
error, so a value that does not parse was read as 0. ListParts with
max-parts=abc then reached the filer with limit 0, which the filer reads
as its default listing limit, and returned every part with
<MaxParts>0</MaxParts>; part-number-marker=abc restarted the listing
from part 1; and ListMultipartUploads with max-uploads=abc answered an
empty, untruncated list even with uploads in progress.
Return the existing InvalidArgument errors (ErrInvalidMaxParts,
ErrInvalidPartNumberMarker, ErrInvalidMaxUploads) instead, the way
max-keys is validated. max-parts and max-uploads are parsed as 32-bit
integers to match the documented 0..2147483647 range.
* fix(volume): reject a negative suffix length in Range headers
parseRange accepted "bytes=--5" as a suffix range of -5 bytes and
returned {start: size+5, length: -5}. The volume server then answered
206 with an empty body and Content-Range "bytes 15-9/10", and the
filer's inline-content reader sliced entry.Content[15:10] and panicked.
Restore the check Go's net/http parseRange has: a suffix-length must be
a non-negative integer (RFC 7233 2.1), so a leading '-' or a negative
value is an invalid range. Apply the same check to the Rust volume
server's parse_range_header.
* test: negative-suffix manifest read is now rejected with 416
The parser change makes a negative suffix an invalid-range error, so
the duplicate-range case in this test can no longer reach the "empty
200" path. Assert the new rejection and that nothing is read.
* volume: treat a zero-length suffix range as unsatisfiable
"bytes=-0" resolved to {start: size, length: 0}, and the range handler
then answered 206 with "Content-Range: bytes 10-9/10", a value clients
must not recombine (and a first-byte-pos RFC 9110 cannot express).
Classify the zero-length suffix with the existing no-overlap case —
its resolved first-byte-pos sits at the content size — so the request
gets a 416 "bytes */size" unless another range in the set overlaps.
Applies to the Go and Rust volume servers alike.
---------
Co-authored-by: chrislusf <chrislusf@gmail.com>
Co-authored-by: user.email <chrislusf@users.noreply.github.com>
* s3: auto-create bucket on POST Object uploads to a missing bucket
PutObject and CreateMultipartUpload lazily create the bucket, but a
signed POST Object upload answered NoSuchBucket. Mirror the PUT path:
when the versioning-state lookup reports the bucket missing, run the
shared handleAutoCreateBucket gate (bucket-name check, AutoCreateBucket
option, admin identity) and then re-read the versioning state. The POST
handler already records the verified signer identity in the request
context, so the admin check and bucket-owner attribution work unchanged.
* test: assert the POST auto-create upload lands, not just non-404
The missing-bucket POST test passed whenever the handler created the
bucket, even if the upload then failed. Drive the full path through the
existing putToFiler fakes — assigned volume accepts the chunk, object
entry and payload land — and assert the 201 success response.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* s3: record verified post-policy signer's account id
The POST policy path authenticates in the handler, after the internal
account header was already scrubbed, so auto-created buckets and the
uploaded objects stored no canonical owner.
* test: assert POST auto-create records bucket owner, keep writes applied in denied case
---------
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* s3: let bucket-scoped Admin grants auto-create matching buckets
autoCreateBucket required a global Admin identity, so an identity
scoped to a prefix like "Admin:outline*" could explicitly CreateBucket
but was denied when an upload auto-created the same bucket. Gate on
CanDo(ACTION_ADMIN, bucket) — the same check explicit CreateBucket
enforces — so a prefix-scoped key lazily creates only matching buckets.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* s3: run auto-create bucket admin through the policy-aware path
A scoped Admin identity with an attached IAM policy that denied
s3:CreateBucket could still auto-create a matching bucket, because
userCanAdminBucket only consulted legacy Actions. Evaluate the same
VerifyActionPermission explicit CreateBucket gets, over a synthesized
PUT /bucket request so the attached-policy check resolves to
s3:CreateBucket on the bucket ARN regardless of the upload's shape.
* s3: record verified identity as owner of auto-created buckets
An admin authenticated outside the middleware (IAM integration, post
policy signature) left no identity in the request context, so an
auto-created bucket stored no owner and never reached the owner index.
---------
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* volume: after losing the leader, dial the leader the other masters name
When the heartbeat stream to the leader breaks and that master stays
unreachable (partitioned or powered off, so dials hang rather than being
refused), the volume server could take close to a minute to register with
the new leader, and the new leader answered assigns with "no volume server
capacity registered yet" in the meantime.
Two things in the retry path added up:
- refreshLeaderFromPeers asked the seed masters in order with up to 5s
each, inside a budget of one pulse (5s by default). With the lost
master listed first, the whole budget went to it and the live peers
that already knew the new leader were never asked.
- Even when the refresh found the new leader, the loop moved on to the
next seed master. When the lost leader had been reached through a
redirect, the next seed can be that same master, so the loop dialed it
again and waited out the 20s gRPC connect timeout.
Ask the lost master last, and when the refresh names a leader other than
the lost master, dial it next.
Assisted-by: Claude <noreply@anthropic.com>
* rust volume: dial the leader the other masters name after losing one
Mirror the Go heartbeat change: refresh_leader_from_peers asks the lost
master last and returns the leader it found, and the loop dials that
leader next instead of a seed that can be the unreachable master itself.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* rust volume: let a named leader consume the next seed slot
Restarting the seed scan on every redirect let an earlier seed that keeps
reporting the dead leader starve every seed after it: the loop bounced
between the redirect target and the next seed forever. Go consumes the
next range value for the redirect instead, so a pass always reaches every
seed. Drop the breaks so new_leader overrides the next slot the same way.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* volume: try a named leader on top of the seed scan
A reported leader used to replace the next seed slot: with seeds A,B
and A persistently naming a dead leader X, X consumed B's slot on
every pass so B was never dialed. The hint is now a one-shot attempt
made before the seed it would have displaced. Its own result stays
pending when a seed attempt names nothing, so a live follower's
redirect chain is followed while every seed is still dialed on every
pass. Mirrors between the Go and Rust volume servers.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* volume: follow a fresh leader hint before the pending seed
A named leader whose stream breaks can leave the peer refresh naming its
replacement; the loop then dialed the pending seed first and paid a dead
master's connect timeout ahead of the fresh name. Fresh hints are now
followed right away, bounded by the seed count and stopping once a name
repeats, so a peer that keeps naming the same dead leader still cannot
starve the seed scan. Go and Rust mirrored.
* volume: dial each named leader once per pass (Go + Rust)
A name that stays pending after the hint chain — the cap was hit, or the
refresh named one already dialed — was retried before every later seed in
the pass: each seed reset the hint budget, so a dead leader still cost a
sleep and a connect timeout per seed. Track the names dialed in a pass,
seeds included, and skip a hint that repeats one.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* volume: let a seed's named leader replace a pending hint (Go + Rust)
The seed dial runs after the hint chain, so the leader it names is the
freshest answer the pass has — but it was dropped whenever a stale hint
was still pending, and the next seed then chased the older name first.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* volume: keep an untried leader hint over the seed's stale answer (Go + Rust)
A seed that keeps naming a stale leader spends the per-pass hint budget
on it; its own answer then overwrote the live name the hint chain had
just reached, so that leader was never dialed. A pending hint now
survives the seed dial until it has been tried.
---------
Co-authored-by: user.email <chrislusf@users.noreply.github.com>
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* security: fail closed on misconfigured gRPC TLS
LoadServerTLS/LoadClientTLS returned plaintext credentials on every
error path once TLS was configured: a typo in tls.min_version or
tls.cipher_suites, cert and key in different directories, or cert/key
set without grpc.ca all left the component serving or dialing
plaintext. A partially configured TLS setup now stops the process at
startup instead; leaving cert/key unset still means no TLS, as before.
The loaders also stat cert/key/ca files up front since the pemfile
providers read lazily and would otherwise only fail at handshake time.
Mirrored in the Rust volume server: partial grpc.volume cert/key, a
missing grpc.ca, or a rustls build failure now abort startup rather
than warn-and-continue on plaintext.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* security: optional server-identity allowlist for gRPC clients
The mTLS client accepts any certificate the configured CA signs, so a
per-workload CA lets one component stand in for any other: a filer
identity can answer as the master. The server side already screens
callers via allowed_commonNames and allowed_wildcard_domain; nothing
screened the server.
LoadClientTLS now wires AdditionalPeerVerification when the caller
configures a server allowlist: grpc.<component>.allowed_server_commonNames
(falling back to a shared grpc.allowed_server_commonNames) matched
against the peer cert's subject CN, DNS SANs, or URI SANs such as a
SPIFFE id, plus a grpc.allowed_server_wildcard_domain suffix match.
Leaving them unset keeps the previous behavior.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* security: URI SANs are exact-match only in the server identity allowlist
A URI identity whose path happens to end in the wildcard domain would
otherwise pass the DNS suffix check.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* security: tighten gRPC TLS fail-closed and server-identity paths
- URI SANs match the server-identity allowlist exactly; the wildcard
domain suffix check applies only to CN and DNS SANs.
- A partially set client_cert/client_key pair is an error when the
cert/key fallback is empty, instead of dialing plaintext.
- Cert/key/ca files are loaded and parsed at startup (LoadX509KeyPair
and CA PEM validation) since pemfile providers read lazily.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* rust volume: error on a partial grpc client_cert/client_key pair
Mirror the Go loadClientTLS path: when client_cert or client_key is
set without its mate and the cert/key fallback is empty, fail instead
of dialing plaintext.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* security: wildcard domain match requires a valid hostname
A CA-signed identity like ".example.com" satisfied the bare suffix check
for wildcard ".example.com" despite not being a hostname. Require a real
subdomain: non-empty label before the suffix, a '.' boundary when the
configured wildcard has no leading dot, and DNS-name syntax. A leading
"*." label still matches so wildcard certs keep working. Applied to both
the Go server/client authenticators and the Rust volume's client-cert
verifier.
* security: ignore a trailing root dot in wildcard domain matching
* tls: reject a partial dedicated client cert/key pair instead of falling back
When exactly one of component.client_cert/client_key is set, warn-and-
fallback silently dialed with the serving identity. Fail closed like the
other TLS misconfigurations in this change; mirrored in the Rust loader.
Regression tests cover a partial pair with a valid fallback.
* rust volume: harden partial client cert pair test with real pem files
* security: verify dialed server identity per target kind on the filer
The filer reused one grpc.filer-verifying dial option for master, volume,
filer and s3 targets, so a broad allowlist accepted the wrong component at
the wrong port and a narrow one rejected legitimate peers. Outbound calls
now pick grpc.<target>.allowed_server_commonNames (falling back to the
client component's and the global list) while still presenting the
grpc.filer client identity.
* security: scope per-target server allowlists under the client component
The filer read grpc.volume.allowed_server_commonNames when dialing a
volume, but that key already denotes the volume's own outbound allowlist
(e.g. the master identity). A deployment restricting volume->master
calls would make the filer reject real volume servers. Per-target policy
now lives under the client component: grpc.filer.volume_*
falls back to grpc.filer.* then grpc.*.
---------
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* s3tables: answer 404 for a table name that was renamed away
A rename leaves the source directory in place to hold the data under
it, with its catalog attributes cleared. UpdateTable, DeleteTable, the
table policy calls and the table maintenance calls read the metadata
attribute of that directory and reported its absence as an internal
error, so they answered 500, and GetTablePolicy answered NoSuchPolicy.
An absent metadata attribute now reads as a missing entry. tableMetadata
reports it as both filer_pb.ErrNotFound and the attribute error, so
callers that test for either keep working. The policy and maintenance
handlers read the attribute through the new readTableMetadataAttr, which
reports filer_pb.ErrNotFound, so each call answers NoSuchTable.
* s3tables: a view record without its metadata reads absent too
* s3tables: refuse a catalog write that lands on a metadata-less record
---------
Co-authored-by: user.email <chrislusf@users.noreply.github.com>
* helm: add commonLabels to the metadata of every object
The chart lets users label pods (podLabels, <component>.podLabels) and
annotate workloads, but offers no way to label the objects themselves.
Backup, ownership and cost policies usually select StatefulSets,
Deployments, Services, PVCs or Secrets by label, and umbrella charts
that set such labels on all their workloads have to patch this chart
to do it.
commonLabels is added to the metadata labels of every object the chart
renders, through one seaweedfs.commonLabels helper. Values may hold
templates, and the whole value may be a template string rendering to a
map, so a parent chart can pass its own labels down. The chart's
app.kubernetes.io/{name,instance,component,managed-by} and helm.sh/chart
keys are dropped, and keys an object already sets (ingress.labels,
httpRoute.labels, monitoring.additionalLabels, grafana_dashboard) win,
so selectors never change and no key renders twice.
Pod templates are left alone (podLabels covers them, and a label change
would roll the pods), as are volumeClaimTemplates, which cannot change
on an existing StatefulSet. checksum/master-config now hashes the master
ConfigMap as rendered without commonLabels, so labels do not roll the
masters either; its value is unchanged.
With commonLabels empty, the default, every template renders exactly as
before. Adds a helm_ci check that labels a release with most components
on and asserts every object carries the labels and nothing else changes.
* helm: do not reject a commonLabels key literally named Error
fromYaml reports parse failures as an {"Error": msg} map, so checking
for that key also matches a user-supplied label of the same name. Only
treat Error as a parse failure when the raw value was not already a
map, so {Error: tracked} renders while non-map input still fails.
* helm: exclude the filer monitoring label from commonLabels
filer.metricsPort adds a hardcoded monitoring: "true" label, which
collided with a monitoring key in commonLabels and rendered a duplicate
map key. Exclude it while metricsPort is set so the object's label wins,
and keep the user's label when it is not.
* helm: accept a templated commonLabels string with an Error key
fromYaml reports a parse failure as a one-key map {Error: "error ..."}, so a
templated string such as 'Error: x' was taken for a parse error. Only fail
when the value is not a map and fromYaml returned its error shape.
* helm ci: keep ConfigMap data in the commonLabels render comparison
Generated Secrets still need normalizing, but the security ConfigMap was
only non-deterministic because of its default jwt signing key. Disable
that key for this render and compare ConfigMap payloads like everything
else so a commonLabels regression in data is caught.
* helm ci: render the security ConfigMap in the commonLabels comparison
Disabling jwtSigning.volumeWrite alone makes securityConfigEnabled false,
so the ConfigMap was absent from the render instead of compared. Enable
mTLS without the generated key so the object is rendered deterministically
and both its labels and data are covered.
* helm ci: strip only generated-credential secret payloads in the commonLabels comparison
* helm ci: assert the effective non-reserved commonLabels
* helm: render integral numeric commonLabels without exponent
---------
Co-authored-by: jeffWild <20067853+jeffWild@users.noreply.github.com>
Co-authored-by: chrislusf <chrislusf@users.noreply.github.com>
Co-authored-by: Chris Lu <chris.lu@gmail.com>
* deps: bump seaweedfs/raft for PreVote
The default raft gains the pre-vote phase so a master returning from a
partition can no longer depose a healthy leader with an inflated term.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* deps: bump seaweedfs/raft for pre-vote review follow-ups
Picks up the post-merge fixes on top of f227dce: per-round vote workers with per-peer dedupe, pre-vote grants counted at a newer term so mixed-version clusters cannot livelock, and state/conf field locking on the startup paths.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* deps: pin seaweedfs/raft to released v1.2.3
PreVote has merged and shipped as raft v1.2.3, so switch off the
PR-branch pseudo-version.
---------
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* helm: add namespaceOverride to deploy into another namespace
When the chart is a subchart of an umbrella chart, every object lands in
the umbrella's release namespace, with no way to give SeaweedFS its own.
namespaceOverride, empty by default, now sets the namespace of every
object through a new seaweedfs.namespace helper, which falls back to the
release namespace. All uses of .Release.Namespace go through it, so the
in-cluster addresses, certificate SANs, ClusterRoleBinding subjects,
HTTPRoute backendRefs and lookups follow the override too. The
self-signed Issuer, the bucket hook Job and the volume resize hook
objects, which had no metadata.namespace, now set it.
With namespaceOverride unset the default render is unchanged. Adds a
helm_ci check that renders with an override and asserts nothing is left
in the release namespace.
* ci: cover the bucket hook Job in the all-in-one namespace test
The all-in-one render set s3.createBuckets, which bucketHookEnabled
ignores in allInOne mode, so the Job never rendered and the namespace
check could not see it. Use allInOne.s3.createBuckets and assert the
Job is present.
* helm: fail on an invalid namespaceOverride instead of reshaping it
Truncating and trimming could silently map an override onto a different
existing namespace and deploy into it. Validate against the DNS-1123
namespace rules and fail the render instead.
* helm: bound the master data volume name at 63 characters
data-<namespace> can exceed the Kubernetes name limit when the release
or override namespace is near 63 characters; truncate the composed name
and trim a trailing hyphen like the other name helpers.
* helm: quote the rendered namespace in YAML fields
* helm: warn that changing namespaceOverride strands claims and data
* helm: say namespaceOverride only moves namespaced objects
* helm: retrigger the PR head sync (no changes)
* helm: warn that secrets and hook resources do not follow namespaceOverride
* helm: warn that hostPath data does not follow namespaceOverride
---------
Co-authored-by: jeffWild <20067853+jeffWild@users.noreply.github.com>
Co-authored-by: chrislusf <chrislusf@users.noreply.github.com>
* volume.copy: reject only on file-count drift, warn on deleted-count
Go warned and kept the copy on any count mismatch. The file count is
exact on both sides of a stable source, so a mismatch means a genuinely
bad copy and should reject; only the deleted count is a rebuild-time
estimate that legitimately drifts. Match the Rust check_copy_counts,
which warns solely on the counter that can drift.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* volume.copy: stage the copy and validate before dropping the old replica
The destination replica was deleted before the copy even started, so a
copy that failed its checks — now including the file-count validation —
left the server with no replica at all. The copy now writes under a
copying/ staging dir the volume loader never scans, the file-size and
count checks run on the staged .idx (LoadCompactNeedleMap gives the same
counters the mount would), and only then is the existing replica deleted
and the staged files renamed into place. A rejected or failed copy
leaves the previous replica untouched, and the .note marker is no longer
needed since partial staged files can never be loaded.
* volume.copy: remove only the files a failed copy moved into place
* volume.copy: keep the .note in-progress marker through the renames
A crash between the staged-file renames left a loadable volume without
its .vif (writable where the source was read-only) or a partial set the
load scan could not find. Write the .note before the renames and land
the .vif first so any partial state is discovered and swept, then drop
the note before the mount.
* volume.copy: serialize copies of the same volume
Two copies of the same volume write the same staged copying/ file names
and can pass the existing-volume check before the first one mounts.
Fail a concurrent second copy as retryable, matching the Rust volume
server's volume_copies_in_flight.
* volume.copy: restore the replaced replica when the swap fails
Once the staged copy validates, the outgoing replica's files are
hard-linked into copying/ before DeleteVolume unlinks them. If a later
rename or the mount then fails, the preserved files are renamed back
and remounted, so a failed swap no longer leaves the node with no
replica. The links are removed on success.
* volume.copy: clear stale needle-map files around the swap
A leftover .ldb/.sdx/.rdb at the destination base is trusted over the
fresh .idx the mount loads, and a mount that fails leaves the backends
it built. Clear them before the staged files land and again when a
failed copy cleans up, gated on no live replica owning them. Also keep
<base>.vif on cleanup while an .ecx index shares the basename.
* volume.copy: skip the swap when the caller has gone
A caller that left during the copy cannot use the result; bail before
the swap deletes the replica it would replace.
* volume.copy: discard a mounted copy when the result send fails
A mounted copy was already announced to the master, so a later failure
deletes it through the store to withdraw the announcement before the
files go away, rather than unlinking files out from under a registered
replica.
* volume.copy: keep a vif-less copy discoverable on a split index dir
With -dir.idx the staged .idx lands outside the data dir, so a copy
whose source has no .vif leaves nothing the volume scan can find and
the mount fails after the replica it replaced was already deleted. An
empty .vif marks the basename; the loader treats it as absent and
writes a real one on mount.
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* volume.copy: preserve the replaced replica on its own disk
* volume.copy: restore the replaced replica's .vif past a kept incoming one
* test: cover the rollback link/restore helpers
* volume.copy: preserve the mounted replica's own files for swap rollback
The swap backup linked files at the copy's target paths, which are wrong
when the replica sits on another disk or keeps its .idx beside the .dat;
a failed swap then restored nothing. Link the mounted volume's real paths
and require every present file to link before the delete, since a partial
backup cannot roll back. Re-check the store at swap time so a replica
that mounted mid-copy is replaced rather than overwritten, and drop a
.kept incoming .vif that occupies the path the replica's own .vif
restores to.
* volume.copy: clear a stale .idx a mount would prefer over the copy
A load switches dirIdx to the data dir when an .idx sits beside the .dat,
so a leftover <data>.idx shadows the fresh index the copy lands in the
idx dir, and a surviving <idx>.idx fills in when the source had no index
at all. Clearing only the derived backends still let a stale needle map
reach the mount.
* volume.copy: leave the shared copying/ dir for racing copies
Copies of different volumes serialize per volume id but share the disk's
copying/ dir; a finishing copy could rmdir it between another copy's
MkdirAll and its first staged write, failing that copy with ENOENT.
* volume.copy: fail replica preservation on stat errors other than not-found
* volume.copy: refuse a mount while that volume is being copied
A copy claims the volume id, then waits on the source with the
replacement staged. A mount in that window loads the replica whose files
the swap then clears; refuse it with Unavailable, retryable once the
copy settles. Mirrors the same refusal on the Rust volume server.
* volume.copy: refuse configure while that volume is being copied
Configure unmounts and remounts; during a copy that already claimed the
volume id the remount loads files the swap then unlinks. Refuse with
Unavailable, retryable once the copy settles. Mirrors the Rust volume
server's same refusal.
* volume.copy: leave a still-standing replica's index on a rejected copy
* volume.copy: preserve an unmounted replica's files for swap rollback
* volume.copy: claim the in-flight id in mount and configure, not just check it
* volume.copy: keep an EC-shared .vif when displacing an unmounted replica
* volume.copy: restore replaced files over a surviving partial
restoreReplacedVolumeFiles deleted a preserved link whenever the base
name was still taken, on the assumption a landed file there belonged to
the copy. When a moved-file removal fails, that left a surviving partial
paired with the restored set's other files — mixed generations in one
replica.
A leftover at the basename now comes off first so the backup goes back,
and the link is kept whenever it cannot go back.
* volume.copy: link a repeated replaced path once so rollback cannot delete it
* volume.copy: refuse allocation and staged adoption under the copy claim
AllocateVolume and adoptStagedVolume still registered a volume while a
copy of it held the destination: NewVolume loads the index before
registration, and the copy then truncates or replaces files the live
replica already opened, leaving it mounted on a stale index.
Both paths now claim the volume id like VolumeMount and VolumeConfigure,
so registration either wins the claim before the copy starts or sees it
and refuses with Unavailable.
* volume.copy: serialize reload loads with the copy claim
A SIGHUP reload ran loadExistingVolume outside the copy's claim: its
.note branch could unlink a copy's files mid-write, and its
NewVolume+SetVolume could register a replica between the copy's recheck
and its swap, leaving a live volume on a stale index.
loadExistingVolume now consults a per-location hook the server installs:
a vid whose claim is held is skipped for the next pass; otherwise the
load takes the claim and its release drops it after registration, so
whichever side claims the id first owns the files for the operation.
* volume.copy: let a claim holder's own mount past the load guard
VolumeMount, VolumeConfigure, VolumeCopy, and adoptStagedVolume all hold
the per-volume claim while calling store.MountVolume, which reaches
loadExistingVolume where the load guard saw the caller's own claim and
skipped the disk: every mount failed with 'volume not found on disk'.
Plumb claimHeld through MountVolume -> LoadVolume -> loadExistingVolume
via the new MountVolumeClaimed entry point so an owner's own load
proceeds while loads without the claim are still skipped.
* volume.copy: keep the strict count check when only deletes drift
Deletes append tombstone rows without moving the file count, so a copy whose source only deleted mid-flight still has an exactly comparable file count. Gate the strict check on file-count stability alone; the deleted-count drift is already a warning inside checkCopyCounts.
* volume.copy: abort the swap when stale index files cannot be removed
A surviving .ldb/.sdx/.idx is trusted over the copied .idx at mount, so
a failed removal must stop the copy before the staged files land, not be
swallowed. Mirrors the Rust side's remove_partial_files abort.
* volume.copy: clear staged leftovers before starting a copy
The staging names are fixed per volume, so a crashed copy can leave
files under copying/; a retry whose new source lacks an optional file
(.vif, or .dat on a remote volume) would rename the previous run's
bytes into place. Clear them up front and refuse the copy when they
cannot be removed.
---------
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* rust volume: remove the redb index when a failed VolumeCopy cleans up
A copy that mounts and then fails validation is unmounted, not deleted. With
a redb needle map the mount has already built <idx base>.rdb, and
cleanup_failed_copy removed .dat/.idx/.vif/.note but not the .rdb, so the
index database outlived the copy. A later copy of the same volume to that
server reuses it (RedbNeedleMap::try_reuse_rdb keeps an .rdb whose recorded
idx size is not larger than the new .idx and replays only the tail), which is
wrong once the source has been compacted in between.
Move the partial-file removal into CopyDestination::remove_partial_files and
include the .rdb, next to the .idx where Volume::file_name puts it.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Et288CnR1mxDyQNdYNMBNq
* rust volume: clear a stale index before a copy writes, and report cleanup failures
A crashed copy, or one whose partial-file removal failed, leaves a stale
.rdb/.sdx next to the .idx; the next copy's mount would reuse it via
try_reuse_rdb's tail replay. remove_partial_files now also covers .sdx
and returns the first non-NotFound error: callers that can still abort
(the pre-copy clear) do so, while the post-failure cleanup warns so a
surviving index is visible instead of silently poisoning the next copy.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* rust volume: restore the pre-copy leftover clear
A leftover index check-in accidentally gated remove_partial_files behind
a constant false, so a stale .rdb/.sdx at the destination survived the
copy that was meant to clear it.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(volume): expect 409 when a read-only upload is refused
x-seaweed-write-outcome: refused-before-append. The HTTP integration
test still expected 500, so Rust unit tests failed on Linux and Windows.
Deletes stay 500.
* rust volume: keep a shared .vif, and clear a stale index beside the .dat
remove_partial_files deleted <data>.vif even when a sealed .ecx still
shares it, undoing the keep in Volume::destroy. It also cleared the
index only under the index base, so a leftover .idx beside the .dat
still made load trust that directory's .rdb.
Skip .vif while a non-empty .ecx is present, and clear both bases when
the index directory differs from the data directory.
* rust volume: do not delete .dat files in the index directory
The index directory can be another location's data directory. Clearing
.dat and .note there removes that replica. Those files come off the
data base only; .idx, .rdb, and .sdx are still cleared on both.
* rust volume: serialize copies of a volume on one server
A second VolumeCopy of the same volume could pass the existing-volume
snapshot before the first copy mounts, then clear or overwrite the
files the first copy still owns. Hold a per-volume claim for the whole
copy, like ec_decodes_in_flight does for decodes, and fail a duplicate
fast so the caller can retry. Mirrors the Go volumeCopiesInFlight
guard.
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* rust volume: keep files when the mounted-copy delete is refused
When a copy mounted and then failed, deleting through the store can be
refused (e.g. the volume is compacting), leaving it registered. The
cleanup then unlinked its files anyway, stranding a live replica. Keep
the files while the volume is still in the store. Mirrors the Go copy
cleanup.
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* rust volume: init volume_copies_in_flight in the http test state
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* rust volume: refuse a mount while that volume is being copied
volume_copy claims the vid, snapshots the store, then waits on the
source. A VolumeMount in that window can load the replica, and the
copy's pre-write cleanup then unlinks its files. Check the claim under
the store write lock and return Unavailable.
* rust volume: refuse configure while that volume is being copied
volume_configure unmounts and remounts. A copy that already snapshotted
the volume as absent can then unlink the files that remount loaded.
Check the copy claim under the store write lock and return Unavailable.
* rust volume: preserve files a copy displaces so a failed copy restores them
* rust volume: keep an interrupted copy's .note with the files it marks
displace_existing_files skipped the .note, so a crashed copy's leftovers
were saved without the marker the load scan rejects them by, and
remove_partial_files then deleted it. A retry that failed restored the
partial set marker-free: after a restart the server could load an
unfinished copy as a normal replica and serve wrong data.
Displace the .note like every other file at the basename so the restore
puts back the exact state the copy found.
* rust volume: restore displaced files over a surviving partial
restore_displaced_files dropped the backup whenever the basename was
still occupied. When the partial cleanup could not remove every file the
copy landed, an existing path just meant the removal failed — deleting
the saved file destroyed the old replica and let the surviving partial
pair with restored siblings of another generation.
Clear the leftover first so the displaced file goes back, and keep the
backup when it cannot: the operator can recover from copying/ rather
than lose the replica outright.
* rust volume: back up an EC-shared .vif the copy truncates in place
A .vif a sealed .ecx shares was left out of the displaced set so the EC
volume keeps it — but the incoming copy opens the same path with
truncate, so a failed copy left the shared .vif overwritten by the
rejected copy's settings with no way back. Go keeps the original via a
hard link because its staged file swaps in by rename; truncating the
shared inode makes a link useless here.
Copy the content into copying/replaced_ like the other backups instead:
the file stays in place for the EC volume during the copy, and the
restore puts the original settings back on failure.
* rust volume: serialize reload loads with the copy claim
A SIGHUP reload scans every volume basename and sweeps .note-marked files
as interrupted copies — including one a live VolumeCopy is still writing.
Mirror Go's loadGuard: the copy-claim set is shared with each disk
location, the scan skips a claimed vid outright and claims any other
through its load, and reconcile_compact_states takes the same skip before
rolling a leftover commit over the copy's files.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* rust volume: hold the load claim through open_volumes
The scan released each claim when its iteration ended, before the queued
volume's index read ran in open_volumes. A VolumeCopy taking the vid in
that gap could displace or remove files out from under Volume::new — the
register the guard exists to prevent. Queued claims now live until the
open pass returns; a second collection of an already-claimed vid merges
into the entry instead of re-claiming.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* rust volume: refuse allocation while that volume is being copied
AllocateVolume registered a volume while a copy of it held the
destination: create_volume loads the index before registration, and the
copy then clears or rewrites files the live replica already opened,
leaving it mounted on a stale index — the window VolumeMount and
VolumeConfigure already refuse.
Claim the volume id like the Go AllocateVolume does, so registration
either wins the claim before the copy starts or sees it and refuses
with Unavailable.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* rust volume: refuse staged adoption while that volume is being copied
The from_staged branch of volume_ec_shards_to_volume renames .copying
files into place and mounts, all while a copy of the same vid could be
rewriting the destination — the same registration window VolumeMount
refuses. Hold the copy claim for the whole adoption, as Go's
adoptStagedVolume does.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* rust volume: drop a partial .note when its write aborts the copy
A write_note failure after displace_existing_files restores the saved replica but leaves the empty or partial marker it just made; the next load scan then deletes the restored replica as an unfinished copy. Remove the marker before restoring, as the Go side's error defer does.
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: user.email <chrislusf@users.noreply.github.com>
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: chrislusf <chris.lu@gmail.com>
* rust volume: keep a copied replica on deleted-count drift, as Go does
check_copy_counts rejected a VolumeCopy whenever the target's deleted count
differed from the source's. On a redb needle map that count is an estimate:
metrics_from_idx rebuilds it at load through a bloom filter (0.1% false
positives per entry, each counted as an overwrite), and the source holds the
estimate from its own last load plus exact runtime updates. A source written
since its load and a fresh load of the same files disagree by a few, so good
copies were refused with "target deleted count [N+1] is not same as origin
deleted count [N]". An index of unique keys 1..=128 already reports one
deletion.
Go fixed the same rejection in #11701 (issue #11678) by warning instead of
failing. Do the same for the deleted count. The file count stays strict: it
is exact on both sides (every non-tombstone .idx row is one file), and the
.dat/.idx size checks still gate integrity.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Et288CnR1mxDyQNdYNMBNq
* test(volume): expect 409 when a read-only upload is refused
x-seaweed-write-outcome: refused-before-append. The HTTP integration
test still expected 500, so Rust unit tests failed on Linux and Windows.
Deletes stay 500.
* rust volume: keep the strict count check when only deletes drift
Deletes append tombstone rows without moving the file count, so a copy whose source only deleted mid-flight still has an exactly comparable file count. Gate the strict check on file-count stability alone; the deleted-count drift is already a warning inside check_copy_counts.
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: user.email <chrislusf@users.noreply.github.com>
* volume.copy: drop the needle-map index a failed copy leaves behind
A VolumeCopy that mounts and then fails validation is only unloaded, not
deleted, so a leveldb (.ldb) or sorted-file (.sdx) index built by the
mount outlives the copy. The next copy of the same volume to this server
then trusts the stale index (generateLevelDbFile replays only past its
watermark), which is wrong once the source has compacted in between.
Mirrors the Rust server's remove_partial_files.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* volume.copy: clear a stale index before a copy writes
A crashed copy, or one whose cleanup failed, leaves a stale .ldb/.sdx/.rdb
next to the .idx that this copy's mount would trust (generateLevelDbFile
replays only past the stored watermark, and a same-size .idx keeps every
un-replayed offset). Abort on a removal error rather than risk a mount
reusing a poisoned index. Mirrors the Rust server's pre-write
remove_partial_files call.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* volume.copy: keep a mounted replica when the final response send fails
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* volume.copy: discard a mounted replica the caller abandoned
A send failure after MountVolume means the caller is gone, but the
replica was already announced to the master. Deleting its files under
the live mount breaks the loaded index; leaving it mounted keeps an
orphan the master routes to. Mirror the Rust copy cleanup: track the
mount, and on error delete through the store (unmount + withdraw the
announcement + remove files, keeping remote data), and check the stream
context before mounting.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* volume.copy: serialize copies of a volume on one server
A second VolumeCopy of the same volume could pass the existing-volume
check before the first copy mounts, then clear or overwrite the files
the first copy still owns. Mirror ecDecodesInFlight: the claim covers
the whole copy, and a duplicate fails fast so the caller can retry.
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* volume.copy: remove only files the copy owns, and every stale index
Three cleanup gaps around which files a copy may remove:
- The failure cleanup ran on any error, so an abort before the first
write (a failed stale-index clear or a failed .note write) could
delete a previous replica's files the copy had not touched. File
removal is now gated on the copy having started writing.
- A mount that finds an .idx beside the .dat builds the needle-map
index under the data base, so leftovers there poison the next copy
the same way the idx-dir ones do. Clear and clean up .idx/.ldb/
.sdx/.rdb under both bases when the dirs differ.
- A regular volume and an EC volume for the same id share <base>.vif.
Keep it while the EC index is on the disk, as the destroy path does.
- When the discard of a mounted copy is refused (e.g. it is compacting)
the volume stays registered; unlinking its files would strand a live
replica.
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* volume.copy: do not unlink a previous replica on an early abort
An abort before the first write now leaves the destination files alone,
but three removal paths still leaked past that gate: a partially-written
.note, and the post-copy defer's .idx/.vif at the data base. Remove the
.note unconditionally (it only ever marks an in-flight copy) and gate the
data-file removals in the outer defer on copyOwnsFiles too.
* volume.copy: refuse a mount or configure while a copy is in flight
A copy claims the volume id, then waits on the source before it clears or
overwrites the destination files. A mount or configure landing in that
window loads the replica whose index files the copy is about to remove.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* volume.copy: abort when a replica registers during the source RPCs
The existing-volume snapshot runs before the copy calls the source, so a
mount that lands in between escapes it. Recheck registration just before
the destructive clear, which can share the index directory with the
mounted replica's bases.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* volume.copy: keep files whenever a replica of the volume stays registered
The cleanup already kept files after a refused delete; a replica a racing
mount registered during the copy needs the same protection, since its
index directory can coincide with the bases being cleaned.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* volume.copy: skip the index cleanup when the copy never picked bases
An early failure leaves the base names empty, so the removals would
otherwise run against ".ldb"/".sdx"/".rdb" in the process directory.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* volume.copy: serialize mount and configure registration under a per-volume guard
The in-flight flag check in VolumeMount and VolumeConfigure was not
atomic with volume registration, and the copy's registration recheck was
not atomic with its stale-index clearing. A mount that passed the check
just before the copy claimed the volume could register after the recheck
and then lose its index files to the clearing.
Replace the bare flag with a per-volume guard: the copy claims it for
the copy's life, mounts and configures hold it across check plus
registration, and the copy holds it across the recheck plus clearing.
Whichever side takes the guard first wins, so a mount either registers
before the recheck (and aborts the copy) or sees the claim and refuses.
Guards persist in the map so a mutex handed to a waiter is never
replaced mid-wait.
* volume.copy: refuse allocation and staged adoption under the copy guard
AllocateVolume and adoptStagedVolume still registered a volume while a
copy of it held the destination: NewVolume loads the index before
registration, and the copy then truncates or replaces files the live
replica already opened, leaving it mounted on a stale index.
Both paths now take the per-volume guard through their registration, so
they either register before the copy's recheck (and make it abort) or
see the claim and refuse with Unavailable.
* volume.copy: serialize reload loads with the per-volume guard
A SIGHUP reload ran loadExistingVolume outside the copy's guard: its
.note branch could unlink a copy's files mid-write, and its
NewVolume+SetVolume could register a replica between the copy's recheck
and its file clearing, leaving a live volume on a stale index.
loadExistingVolume now consults a per-location hook the server installs:
a vid with a copy in flight (or whose guard the copy currently holds) is
skipped for the next pass; otherwise the returned release keeps the
guard held through registration, so the copy's recheck either sees the
volume and aborts or clears first.
* volume.copy: skip compaction-state reconcile while a copy owns the volume
* volume.copy: let a guarded operation's own mount past the load guard
VolumeMount, VolumeConfigure, adoptStagedVolume hold the volume guard
locked, and VolumeCopy holds copying=true, while calling store.MountVolume.
The mount reaches loadExistingVolume where the new load guard saw the
caller's own hold (TryLock fails, or copying set) and skipped the disk:
every guarded mount failed 'volume not found on disk' — a completed copy
deleted its copied files, configure left the volume unmounted, staged
adoption failed after renaming.
Plumb ownLoad through MountVolume -> LoadVolume -> loadExistingVolume via
the new MountVolumeClaimed entry point so the owner's own load proceeds
while unrelated loads are still skipped.
* volume.copy: fix own-load test call, assert unguarded mounts still skip
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* volume.copy: refuse unmount and delete while a copy owns the volume
---------
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* filer.backup: add -s3FlattenVersions to mirror the current S3 view
Versioned buckets are stored as <key>.versions/v_<versionId> files plus a
latest-version pointer on the .versions directory, so a backup mirror held
internal version files and could never be restored through the S3 API.
With -s3FlattenVersions the backup maps events on .versions directories
onto the flattened object key: a new latest version uploads to <key>, a
latest delete marker or a cleared pointer deletes <key> when
-doDeleteFiles is set, and the suspended-versioning null object resolves
to the regular path. Version files themselves are never mirrored. The
initial snapshot seeds only the version the pointer names.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* filer.backup: serialize flattened writes per object key
Same-path non-barrier dir updates run concurrently in the metadata
processor, so two pointer updates for one object -- or a pointer update
and a regular-path write to it -- could overlap and let the older upload
finish last. Lock on the flattened key: versions events lock the object
key, every other event its own path, via a striped mutex.
* filer.backup: apply exclusions to the flattened object key
File-name and path excludes ran against the .versions entry, so an
excluded object was still copied flat and its deletion still removed the
destination copy. Match the rules against the flattened key on both the
apply and the versions-dir-removal paths.
* filer.backup: re-resolve the pointer on version-file events
A tag or attribute update on the current version touches the v_* file,
not the .versions pointer, so dropping every version-file event left the
mirror with stale tags until the next pointer flip. Re-resolve the
pointer on each version-file event; noncurrent versions still stay out.
* filer.backup: tolerate vanished versions dirs in flattened snapshot
The flattened seed path returned source errors before the existing
missing-source policy ran, so one version deleted mid-walk could force a
full re-walk. Apply the same 404/superseded skip as the generic seed
branch, and cover the review fixes with tests.
* filer.backup: lock both flattened keys an event touches
A rename or delete can retire one flattened key while writing another,
so serialize on the old- and new-side keys together.
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* filer.backup: refresh the flattened key only for the still-current child
A v_* event on a non-pointed version cannot move the flattened key, so
skip the re-upload; tag and attribute updates on the current version
still flow through.
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* replication: let resolved-latest writes bypass the source staleness check
A filer sink skips a CreateEntry whose source path no longer holds the
entry, but a flattened write lands on a key whose source path is not the
one it was read from, so the check dropped every flattened upload.
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* filer.backup: reclaim destination chunks on flattened deletes
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* filer.backup: route regular-path events for versioned keys through the pointer
A null-object write or delete at the regular path can coexist with a
.versions pointer that names a live version; the pointer decides the
flattened key, so deleting the null object must not remove it.
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* filer.backup: make the checkpoint identity cover the flattening mode
Resuming a checkpoint taken in the other mode would skip the flattened
re-seed and leave mirrored version files behind; a mode change now
starts a fresh snapshot.
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test: cover flattened regular-path routing and per-mode checkpoints
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* filer.backup: resolve both sides of a flattened rename
* test: cover flattened renames with one versioned side
* replication: let resolved-latest writes overwrite a newer destination
The stale-destination skip trusted a newer destination mtime, but a
resolved-latest write can legitimately install an older version over the
copy of a since-deleted latest. Skip only on provably identical content
(matching chunk provenance, or equal ETag with full byte coverage).
* filer.backup: rescan .versions when the latest null object is gone
Deleting ?versionId=null removes the regular-path object without touching
the null-is-latest pointer, so the flattened key kept its stale copy.
Fall back to the newest remaining version (or a flat delete), the same
rescan the S3 read path applies.
* replication: compare inline and remote content in the destination-holds checks
A chunk-less entry keeps its bytes in Content and a remote-only entry in
the object RemoteEntry points at, but the identical-content shortcuts
only compared chunk lists and chunk-derived ETags. Two different inline
files looked identical, so a flattened re-copy or a same-mtime rewrite
kept the previous version's bytes. Compare Content bytes and remote
object identity wherever the checks claim the destination already holds
the reference's content.
* filer.backup: paginate the null-latest .versions rescan
A limit=0 listing is capped at DirListingLimit server-side, and
old-format version names sort oldest-first, so a history longer than one
page left the flattened copy on an older version after the null object
was removed. Page through the directory like the S3 read path does.
* replication: require matching writable metadata for resolved-latest skips
A skip keeps the destination's stored metadata, so identical bytes were not
enough: stale attributes or extended entries (e.g. a superseded version id)
survived under equal content. Compare the fields a rewrite controls —
attributes minus filer-assigned inode/crtime, with the store's atime/ctime
zero-defaults applied — plus the extended map, in both the provenance and
ETag fast paths.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* replication: require the extended key to exist when comparing metadata
* replication: ignore a stored rule TTL when the resolved write carries none
* filer.backup: rename -s3FlattenVersions to -s3.flattenVersions
Match the -s3.* flag naming convention used by the other s3 options.
---------
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* s3sink: upload through a concurrent-safe ReaderAt so parts do not thrash
s3manager splits a seekable body into parallel section readers, and every
part's ReadAt landed on the single chunk buffer in ChunkStreamReader: parts
at different offsets kept evicting each other's chunk and refetching it
whole, which produced massive read amplification against the volume
servers and collapsed throughput.
Build the upload body from filer.ChunkReadAt instead, the same
concurrent-safe reader the mount and query paths use: each part's range is
served from a shared reader cache with parallel chunk fetches and
range-reads for clipped views, so every byte is fetched once per pass.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* filer: buffer recently fetched chunks for concurrent readers
* Revert "s3sink: upload through a concurrent-safe ReaderAt so parts do not thrash"
This reverts commit be1ec2d7c7.
---------
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* seaweed-volume: pin the read-only write status to 409 in the test
PR #12236 moved read-only upload refusals from 500 to CONFLICT with the
refused-before-append proof; the test still asserted Go's 500 for the
write. Deletes on a read-only volume keep surfacing 500.
* seaweed-volume: correct the read-only status comment in the test
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* s3tables: refuse a create that a namespace delete can lose
DeleteNamespace checks the namespace for children and then deletes its
directory recursively in a second filer call. The filer locks each path
on its own, so a table created under the namespace between the two is
not seen by the check and is removed by the delete, while both requests
answer 200. The mirror case loses the table too: a create that passed
its namespace check before the delete finished re-created the namespace
directory through the filer's parent creation and recorded a table in a
namespace that no longer exists.
Each side now writes before it reads. DeleteNamespace claims the
namespace record with a fresh token before it scans, deletes only while
the record still holds that claim, and lifts its own claim when it
stops. CreateTable, RegisterTable and CreateView re-read the namespace
after writing their record and back out when it is claimed or gone:
they remove what they wrote and answer 409 or 404 instead of reporting
an entry that does not survive, or 500 when the record could not be
removed and may remain. An entry the scan missed was written after the
claim, so its create sees the claim; a create and a delete can both
fail, but no longer both succeed. A create is also refused up front
while a claim is held, but only after its authorization check, so a
caller that may not create there does not learn of the delete.
A create writes its record marked "create" and lifts the mark once the
namespace is confirmed open. Until then GetTable, GetView, UpdateTable
and a second create treat the record as absent, so no request is told
about a table or view that the first create may still back out.
* s3tables: keep a create's unfinished record out of reads and listings
* s3tables: refuse catalog writes on a record an operation claimed
* s3tables: answer 404 when a create's write path loses the namespace
* s3tables: confirm the rename destination's namespace like a create
* s3tables: cover unfinished-record reads, claimed-record writes, rename claim
* s3tables: lift a leftover claim when a no-token delete clears the record
* s3tables: answer 409 when a claimed record turns away a write
UpdateView, TagResource, and UntagResource surfaced ErrConcurrentUpdate
as a 500, so an expected conflict looked like a server fault to the
caller. Map it to Conflict the way UpdateTable and the policy handlers
already do.
* s3tables: name the lost source when a rename finds it deleted
* s3tables: report an entry deleted before its update as not found
The filer answers an UpdateEntry on an entry removed since the lookup
with codes.NotFound. mutateEntryExtended returned that error unchanged
instead of filer_pb.ErrNotFound, so a rename whose source was deleted
mid-claim answered 500 instead of NoSuchTable or NoSuchView.
MemFiler now answers a missing entry the same way, so the rename tests
for tables and views take the real filer's path.
* s3tables: unpublish the destination when a rename's source clear fails
* s3tables: refuse to delete a record another operation claimed
* s3tables: release catalog claims on an uncanceled cleanup context
* s3tables: answer 404 when a racing delete already removed the table
* s3tables: keep the rename destination marked until the source clears
* s3tables: guard the rename rollback against a recreated source and an unfinished destination
* s3tables: drop the unfinished destination only once the source is restored
* s3tables: re-resolve the purge target on the record a delete claims
* s3tables: cover a delete that meets a moved metadata location
* s3tables: answer 409 when a create's record changed under it
* s3tables: adopt a record still marked by its unfinished create
* s3tables: skip unfinished creates in the namespace empty check
* s3tables: delete only the record the caller was authorized on
DeleteView and DeleteTable check permissions on the record they read
first, then read the name again to claim it. A record that took the name
in between, such as a table or another owner's view created after a
concurrent delete, was claimed and removed on the first record's
authorization, and DeleteView never compared the caller's version token
with it.
The second read now repeats the first one's checks: a record of the
other kind or an unfinished create answers 404, and a token that no
longer matches answers 409. A delete without a token also compares the
record's version token with the authorized one and answers 409 when they
differ.
* s3tables: re-check the parent after a create lifts its mark
A create that passed checkParentOpen and paused before lifting its mark
could still confirm while a namespace delete held the claim: the delete
scans the still-marked record, skips it, and removes it under a create
that already answered success. confirmParent now re-reads the same
parent record after the mark clears and backs the confirmed record out
when the namespace moved on.
* s3tables: mark the rename destination with a claim of its own
claimCatalogEntry adopted every create-marked record, including a
rename's destination: a create landing after the source cleared could
replace the destination before the rename published it, and with the
source name reused the restore could leave the table on neither name.
The destination now marks with the rename's own claim, which readers
still treat as unpublished but a create must refuse, and the commit
re-checks the destination namespace after the record goes live. The
source keeps its rename claim until the commit lands so a namespace
delete scanning the cleared record still refuses.
* s3tables: compare the parent's createdAt, not its editable metadata
A namespace update rewrites ExtendedKeyMetadata in place, so a create or
rename whose second parent read landed across an UpdateNamespace saw a
changed blob and reported the still-live namespace as gone: creates backed
their record out and answered 404, renames undid their destination.
createdAt survives the edit but changes on a delete+recreate, so it still
identifies the record.
* s3tables: retry lifting the rename's claim off the cleared source
The release's only failure mode is a transient filer error — nothing else
can write the cleared record — yet a single loss left the claim behind:
the old name stayed refused to creates and the namespace counted the
record as a child. Retry the release before warning.
* s3tables: page the namespace scan before deleting it
A page of skippable entries — bare directories or unfinished creates —
ended the scan with hasChildren false, so a table on a later page was
removed with the namespace. Keep listing with StartFromFileName until a
child shows or the listing ends.
---------
Co-authored-by: chrislusf <chrislusf@users.noreply.github.com>
* test: add s3 integration tests for SQLite Cloud Backed Storage (#12235)
SQLite CBS stores a database as named objects in a bucket and relies on
conditional PUT — put-if-absent for new blocks and ETag compare-and-swap
on the manifest — so concurrent clients cannot corrupt the database.
The suite runs a weed mini cluster (conditions are evaluated under the
object's write lock on the owner filer via the routed-write path) and
covers:
- the CBS storage primitives: create/put/fetch/list/delete/destroy
- If-None-Match: * put-if-absent, success and 412 rejection
- If-Match ETag compare-and-swap, including stale-ETag rejection and
the re-read/retry loop CBS clients run on 412, on plain and
versioned buckets
- racing put-if-absent and manifest CAS writers: exactly one commit
wins, every update survives
- conditional CompleteMultipartUpload, sequential and raced
- an end-to-end bcv-style upload, download-and-verify, and checkpoint
A matching job is added to the S3 mutation regression workflow.
* test/s3: drop issue references from the sqlite cbs suite comments
* test/s3: abort every losing upload after the conditional multipart race
* test/s3: assert the winning writer's body after the conditional write races
* test/s3: spell out the get-returned etag in the cbs suite
* test/s3: assert the completed multipart ETag the cbs suite feeds back
The conditional-completion checks pass completeOut.ETag into If-Match;
if the response carried no ETag the check would silently run
unconditional.
* test/s3: drop the unused generation parameter from the cbs block body
* test/s3vectors: data-plane lifecycle integration test
TestVectorDataPlaneLifecycle spawns a real weed-vector against the lance
listener, mirrors the pytest cluster's flags, waits for the gateway to
discover the registered engine, and walks the full lifecycle: create
bucket and index, put/get/list vectors, a filtered query that excludes the
nearer non-matching vector, delete vectors, index, and bucket. The test
skips when no engine binary resolves (VECTOR_BINARY, ./weed-vector, PATH).
* test/s3vectors: fail fast when the engine dies before listening
* test/s3vectors: harden engine launch and tighten lifecycle assertions
* test/s3vectors: wait on the engine's exit once through a broadcast channel
* s3vectors: record request metrics on the dispatch path
The S3 API tracks every operation through S3RequestHistogram,
S3RequestCounter, and S3InFlightRequestsGauge; the vectors dispatch rode
the same listener with no instrumentation. The op name labels each
series under s3vectors.<Operation>.
* s3vectors: count auth rejects and keep bucketless metrics alive
* s3vectors: bound GetVectors and DeleteVectors key lists
Both forwarded req.Keys to the engine unbounded; AWS caps these at 500,
the same window a PutVectors batch shares.
* s3vectors: cap GetVectors at the AWS 100-key limit
GetVectors allows at most 100 keys per request while DeleteVectors
allows 500, so give vectorKeysBounds the per-call maximum instead of
sharing the put-batch window.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* s3vectors: assert out-of-window key lists never reach the engine
Record get/delete calls on the fake engine and cover both handlers
rejecting empty and oversized lists with 400 ValidationException.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
filer.data, filer.logs and master.logs accept type existingClaim and render
the pod volume for it, but the volumeMount (and, for logs, -logdir) only
covered hostPath, persistentVolumeClaim and emptyDir. With
filer.data.type=existingClaim the claim was attached to the pod but never
mounted, so the filer kept its metadata store on the container's own
filesystem and lost it on every restart. admin and worker already include
existingClaim in the same conditions.
Adds a helm_ci check that renders both StatefulSets with existingClaim and
asserts the mounts and -logdir.
Co-authored-by: Pat White <5015863+patwhite@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* webdav: answer 204 to a PUT that replaces a file
* webdav: decide the 204 from what OpenFile actually replaced
A Stat before the write left a window where a delete or create in
between made the status wrong, and it cost a second filer lookup. The
OpenFile create path already knows whether it removed an existing
entry, so it now marks the request context and the writer maps 201 to
204 only on a real replace.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test/webdav: cover OpenFile's own put-outcome flag for both create paths
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* webdav: take the put outcome from the create's OldEntry
A stat-then-create still races: a second writer landing between the
check and the write makes the reported status describe a state the
write never saw. The filer decides create-vs-replace under its per-path
lock and reports the previous entry in the response's metadata event,
so flag the PUT replace from OldEntry instead of a pre-write stat. The
in-place update keeps the replaced entry's crtime and inode and reuses
the filer's own chunk cleanup, so the separate removeAll goes away, and
an exclusive create now defers to the filer's atomic O_EXCL instead of
a racy local check.
* webdav: send no body with the 204 a replacing PUT answers
---------
Co-authored-by: user.email <chrislusf@users.noreply.github.com>
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* volume server: add idx_content to VolumeIncrementalCopyResponse
A remote-tier delete is an index-only tombstone row with no .dat record,
so it cannot reach an incremental copy through the .dat stream. Carrying
the .idx tail verbatim lets the backup reproduce those rows at their
exact positions.
* backup: carry index-only tombstones through incremental copies
A remote-tier delete leaves an offset-0 tombstone in .idx with no .dat
record. The copy only streamed .dat bytes and regenerated .idx from
them, so the tombstone never reached the backup: the deleted needle
stayed live and every later incremental diverged and rebuilt again.
backupCursor and ResumeByLastEntry now accept an index-only tombstone
as a boundary, matched by key and size since there is no record to read
a cookie from. VolumeIncrementalCopy appends the .idx tail after the
.dat tail, and IncrementalBackup writes it at the index boundary,
truncates, and loads the new rows — so tombstones land at the exact
positions the replica has them.
* seaweed-volume: mirror index-only tombstone handling in incremental copy
resume_by_last_entry accepts an offset-0 tombstone boundary matched on
key and size, skips its missing cookie check, and resumes .dat streaming
at the next entry that has a record — index-only rows ride the new
idx_content stream instead of reporting divergence. The server captures
the .idx bound before the .dat one (a row is published only after its
bytes) and streams the tail after the data.
* backup: keep incremental copy bounds and streams consistent
Review fixes:
* Bound capture moved after the resume reads and reads .dat/.idx sizes
together (Go FileStat, Rust file_stat under one lock), so every row
resume saw is inside the index bound and each row's record is always
inside the data bound.
* A compaction commit changes CompactionRevision: the server now rejects
the copy when the revision moved between resume, file opens, or the
end of the stream, so a backup never mixes revisions. The .idx is
opened up front to pin the inode.
* An index tail that ends before the captured bound is an error, not a
silent success.
* IncrementalBackup rolls the .idx tail and .dat bytes back to the
pre-copy boundary when the stream fails, so a partial row cannot wedge
backupCursor and unapplied rows cannot pass for current ones.
* The index tail is replayed through a small indexTailReceiver interface,
covering the leveldb needle maps too, not just the in-memory map.
* backup: bound the resume scan to the captured index size
The resume and the stream bounds are now captured in one order:
revision, then the .dat/.idx pair, then the scan. ResumeByLastEntry and
resume_by_last_entry take the index bound and scan no further, so a row
appended while the copy is being set up can never enter the index
stream without its .dat bytes.
The index-tail replay stays on the in-memory map only: the leveldb
map's DoOffsetLoading writes to the .cpldb compaction database, not the
live one, so it cannot serve a live tail replay; other maps keep
regenerating the index from .dat as before.
* seaweed-volume: guard incremental copy total against a located offset past .dat
A corrupt index can place the resume offset beyond the captured .dat
size; has_data already refuses the reader, but total still evaluates
dat_size - s and underflows.
* backup: bound the incremental .dat tail by index-covered records
A record staged but not yet indexed can sit inside the captured .dat
bound while its row stays outside the index bound — the fsync gap
between staging and publishing is not covered by one stat. Streaming
those bytes leaves them unindexed on the backup, and the next run
appends the record again. Bound the .dat tail by the last record the
streamed rows cover on both producers, and cover the bound walk with
tests.
* backup: cap the .dat stream at the captured bound and roll back replay failures
sendFileContent could read past stopOffset on its last chunk, so a
needle appended after FileStat arrived without its index row and the
next copy appended at the wrong position. Each read is now limited to
the remaining captured bytes.
A failed truncateIndex or DoOffsetLoading after a successful stream
left the index tail on disk unapplied, which a later backupCursor could
mistake for applied rows; both now roll the tail back to the pre-copy
boundary like the stream-error path.
* backup: bound the incremental .dat tail by the maximum covered offset
A key-sorted .idx ends on the highest-key row, which need not be the
record physically last in .dat, so the last-row bound could cut off
live records. Take the end of the highest-offset row inside the bound
instead, and cover the case with a sorted-idx test in Go and Rust.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* backup: restore .dat and the needle map when index replay fails
A failed DoOffsetLoading left the received .dat bytes in place and could
leave partial mappings for rows the index rollback just removed, so the
next copy appended the same records again or read stale entries. Truncate
.dat back to the copy boundary and restore the touched mappings and
metrics from a pre-replay snapshot.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* backup: cover no index rows with the superblock bound
When no row inside the index bound names a record — an empty index or a
bound of index-only tombstones — the covered end is the superblock, not
the captured .dat size, so staged bytes are never streamed without a
covering row.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* seaweed-volume: apply the covered bound even when the index streams no rows
The bound was gated on idx_end > idx_start, so an empty captured index
skipped it entirely and a staged .dat tail still shipped. Gate on the
stream-index predicate instead, matching Go's streamIndex check.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* backup: propagate index coverage read errors
* backup: rebuild the needle map from the truncated index on replay failure
* seaweed-volume: propagate index coverage read errors
---------
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* webdav: overwrite on MOVE when the Overwrite header is missing
* webdav: defer a MOVE destination's delete into the rename
x/net removes an existing destination before Rename, so a refused or
failed move — a CanRename rejection, a vanished parent, a transient
filer error — loses the target entirely. For an overwriting MOVE onto
an existing destination, run the request against a filesystem that
skips the destination RemoveAll and lets the filer's rename replace the
target inside one transaction instead; a file vs. collection clash
still takes the delete + rename path. Rename itself now tolerates an
existing target, which the atomic filer rename already handles.
* webdav: only default Overwrite when the header has no values
An Overwrite header sent with an empty first value makes Header.Get
return "", which the defaulting treated as absent and rewrote to T —
overriding a client's explicit F in a later value. Check the value list
instead so a present-but-empty header is left alone.
* webdav: move a collection onto a non-empty one via delete + rename
deferredDeleteFS skips the destination RemoveAll so the filer rename can
replace the target in one transaction, but the filer only replaces an
empty collection: moveSelfEntry deletes the target non-recursively and
refuses ErrNonEmptyFolder. Detect a non-empty collection destination and
take the depth-infinity delete + rename path moveFiles would have run.
* webdav: park a clashing MOVE target aside until the rename lands
A refused rename (cross-collection move, bucket target) after deleting
the destination lost its contents; the target now renames to a hidden
sibling first and is restored when the winning rename fails.
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* webdav: restore a parked MOVE target under a detached context
A client disconnect cancels the request context mid-move, which also
canceled the restore and stranded the destination under its hidden
sibling name; the restore and parked-entry removal now run under
context.WithoutCancel with a bounded deadline.
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* webdav: bound the parked MOVE target's restore after the move
The detached restore deadline started before the source rename, so a
long move could exhaust it; each detached context now starts where its
work begins. A failed parked-entry removal retries briefly and its
warning names the parked sibling for recovery.
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* webdav: restore a parked MOVE target without replacing a new claimant
A client can create the destination name while the old target is parked, so the restore rename must fail instead of replacing that new entry.
* webdav: propagate RemoveAll's context to the filer delete
The parked-target cleanup runs under a bounded detached context, but the delete RPC discarded it; passing ctx through makes the deadline effective.
---------
Co-authored-by: user.email <chrislusf@users.noreply.github.com>
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test: move the s3vectors pytest suite under test/s3vectors
The vectors API tests lived in test/vectors next to the new s3vectors
integration home; the rust-vector-tests workflow's paths and working
directory move with it.
* gitignore: cover the binaries the s3vectors Makefile builds
* test/s3vectors: integration test design and catalog-surface suite
The design doc lays out a two-tier suite: catalog/policy ops that need no
vector engine run against a real weed server over HTTP, while the data
plane skips itself when no engine is reachable. The first test brings up
the server and exercises the bucket/policy lifecycle, encryption-config
round-trip, and pre-engine validation boundaries.
* test/s3vectors: sign API calls so requests reach the vectors router
The gateway selects the vectors router off the SigV4 credential scope —
unsigned requests with only X-Amz-Target fall through to the plain S3
router, so every call in the suite failed before dispatch. Sign each
request with the s3vectors service name and point the spawned server at
a written identity file so the signature is actually verified.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test/s3vectors: bound the spawned server's ports, listeners, and clients
weed server derives each gRPC listener as HTTP+10000, so ephemeral
allocation could hand back a port whose pair is invalid or taken; the
-s3 defaults also bind the Iceberg and Lance listeners. Allocate the
HTTP/gRPC pairs together, pin -ip/-ip.bind, disable the extra listeners,
and give the readiness probe and API client deadlines.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test/s3vectors: verify policy round-trip and gate the engine skip on 503
The stored policy check accepted any JSON body, and a delete that left
the document behind could not be caught — compare the returned document
verbatim and assert a follow-up Get returns 404. The engine probe never
created its bucket, so a healthy engine still produced a skip; create it
first, skip only on a confirmed 503, and fail on any other error, then
exercise the catalog-side index ops when an engine answers.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
s3: AbortMultipartUpload answers NoSuchUpload for an upload that no longer exists
Complete and abort both remove .uploads/<id>. abortMultipartUpload answered
204 whenever that record was missing, so aborting an upload that had already
completed reported success while the object stayed in place, and a client
could read it as the object being discarded. AWS returns 404 NoSuchUpload
("The specified multipart upload does not exist") for that case, and
ListParts here already does.
Answer ErrNoSuchUpload when the record is not found. checkUploadStillOpen
reuses abort for cleanup after the upload is gone; NoSuchUpload is not a
failed cleanup there.
The leftover-directory path from #11385 is unchanged: a .uploads/<id> that
outlived its completed object is still removed metadata-only and answered.
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* volume server: add last-entry cursor fields to VolumeIncrementalCopy
* backup: resume incremental copies by index position instead of appendAtNs
appendAtNs is stamped by whichever replica received the write, so the
same needle carries a different timestamp on every replica. A backup
that resumes from a different replica than last time can append the
last needle again or skip needles written inside the clock gap.
Send the local copy's index entry count and last entry identity (key,
size, cookie) with VolumeIncrementalCopy. The replica verifies the same
record sits at that position and streams from the next index entry, so
the resume no longer depends on any replica's clock. When the boundary
record does not match, or the local copy is ahead, the replica returns
FailedPrecondition and the backup rebuilds the local copy from scratch.
Old servers ignore the new fields and keep the timestamp behavior.
* rust volume server: mirror positional resume in VolumeIncrementalCopy
* volume server: add entries_hash prefix checksum to VolumeIncrementalCopy
* backup: verify the whole index prefix, not just the boundary entry
* rust volume server: verify the index prefix hash in resume_by_last_entry
* backup: keep the source's compaction revision on rebuilt copies
* backup: diverge on a zero-offset boundary entry
A remote-tier tombstone lives in the .idx with no .dat record, so
readNeedleAt(0) parses the superblock and its error bypassed the
ErrIncrementalCopyDiverged mapping; the backup got a generic failure
instead of FailedPrecondition and never rebuilt.
* rust volume server: mirror zero-offset boundary divergence
* rust volume server: gate the empty-dat early return on entry_count
With a positional resume the Go handler still reaches ResumeByLastEntry,
so an empty replica reports divergence; the Rust early return answered a
clean empty stream and weed backup kept a stale copy.
* backup: hash the index entry's size field, not trailing offset bytes
In a 5BytesOffset build the offset occupies bytes 8-12 and the size
starts at byte 13, so the hash folded in one byte of the offset.
Matching records at different offsets then hashed differently and a
valid backup was reported diverged and rebuilt.
* backup: read the index in whole-entry chunks for the prefix hash
When entries are 17 bytes a 1 MiB read buffer does not end on an entry
boundary: the partial tail entry was skipped by the loop but still
counted as consumed, so every later chunk was hashed at a misaligned
position.
* rust volume server: hash the index entry's size field after the offset
Mirrors the Go hashIndexEntries change: with the 5bytes feature the
offset occupies bytes 8-12, so starting the size slice at byte 12
folded one offset byte into the hash and matching records at different
offsets compared diverged.
* rust volume server: read the index in whole-entry chunks for the prefix hash
Mirrors the Go change. With 17-byte entries a 1 MiB buffer does not
end on an entry boundary, so slicing the partial tail entry panicked
once a backup held 61681 entries.
* rust volume server: default the new incremental copy request fields in tests
* test: align vector bucket policy coverage with vector actions
* s3vectors: validate ARN service and resource shape before resolving indexes
* test: exercise signed vector bucket policies and catalog lifecycle
* ci: require the expanded vector integration matrix and all gateway tests
* test: reject index ARNs in bucket-level API requests
* s3vectors: scope ARN resolution to the gateway namespace
* s3vectors: reject name fields that conflict with resolved ARNs
bucketARNs and resolveIndexARNs already reject an indexArn that names a
different bucket than vectorBucketArn, but a vectorBucketName or
indexName disagreeing with the ARNs was silently dropped. Validate the
plain names against the resolved ARN values the same way.
* test: cover conflicting vector names and ARNs in live requests
---------
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* s3vectors: reject empty boolean-operator lists and sub-objects in filters
An empty $and/$or list or a nested empty object rendered '()'' — invalid
SQL surfaced as an engine error instead of a clean ValidationException.
* s3vectors: e2e conformance for vector bucket policy evaluation
The handler-level test covers statements; this runs the whole stack a
client sees — HTTP dispatch, authentication, policy evaluation, catalog,
engine — and walks the full lifecycle: a non-owner is denied without a
grant, an Allow on the bucket ARN grants every index under it, a Deny
vetoes an Allow on the same action, and deleting the policy restores
owner-only access.
* s3vectors: prove deny precedence and grant removal in policy e2e
* s3vectors: reject field operator objects with no operators
* test: use an s3vectors action in the vector policy round-trip
* s3vectors: reject empty boolean-operator lists and sub-objects in filters
An empty $and/$or list or a nested empty object rendered '()'' — invalid
SQL surfaced as an engine error instead of a clean ValidationException.
* s3vectors: reject field operator objects with no operators
* s3api: label bucket-level requests correctly in the access log
getResourceType treated a request as bucket-level only when the object key
was "/". NormalizeObjectKey strips the leading slash, so bucket routes arrive
with an empty key and every bucket operation was logged as an object
operation: ListObjectsV2 as REST.GET.OBJECT, DeleteObjects as
REST.POST.OBJECT, and the lifecycle, ACL and policy calls likewise.
Treat an empty object key as bucket-level too. The new test covers five
bucket-level and two object-level request shapes; it fails without the fix.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
* s3api: label only POST ?delete as a batch delete in the access log
With bucket-level requests now reaching the bucket branch, any method on
/bucket?delete was labelled BATCH.DELETE.OBJECT. Only the POST route is
DeleteObjects; label other methods as the plain bucket operation.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
* mount: refuse to attach opens to a draining file handle
With the writeback cache, releasing the last open of a file leaves the
shared FileHandle in the map at counter zero while an async flush
drains it; the worker removes the map entries once the flush commits.
AcquireHandle waits for a pending flush and then attaches, but the two
steps are not atomic: a release can slip in between, drop the counter
to zero, and queue the drain while the opener goes on to mint a new
file-handle id on the draining handle. When the drain completes,
RemoveFileHandle deletes every fh id for the inode -- including the one
just minted. The kernel still holds that id, so its next FUSE_READ
gets ENOENT from GetHandle; the failed read leaves the folio not
uptodate and the kernel surfaces EIO to userspace. This was the flaky
TestWritebackCacheConcurrentMixedOps failure.
AcquireFileHandle now refuses to attach when the shared handle has
reached a zero counter and reports it as draining. AcquireHandle and
the Create path retry the whole wait-resolve-attach sequence; the next
pass blocks on the pending flush (flag registration and the counter
drop share pendingAsyncFlushMu) and builds a fresh handle from the
post-flush entry. The DLM-failure release in AcquireHandle also goes
through ReleaseHandle so it cannot strand a counter-zero handle
outside the drain bookkeeping.
Removes the refreshEntryForRead retry from #12206, which worked around
the symptom by refetching the entry after the fh id was already gone.
Test: TestAcquireFileHandleRefusesDrainingHandle covers the refusal
and fresh-handle path; TestWritebackCacheConcurrentMixedOps and an
amplified concurrent-overwrite repro run clean.
* mount: keep refresh-on-error retry for stale-chunk reads
* s3vectors: bound ListVectorBuckets and ListIndexes maxResults
Both pass maxResults to the catalog unbounded; AWS caps these list pages
at 100, the same way ListVectors is bounded and topK is bounded.
* s3vectors: raise ListVectorBuckets/ListIndexes maxResults cap to 500
AWS documents a valid range of 1-500 for both list APIs; the shared
limit of 100 rejected legitimate page sizes.
* s3vectors: reject explicit maxResults of zero on list calls
AWS requires maxResults >= 1, but a plain int field cannot tell an
omitted field from an explicit zero. Making MaxResults a pointer keeps
the catalog default for omitted fields while a supplied zero fails
validation.
* s3vectors: test max ListIndexes page against an existing bucket
* s3tables: reject s3vectors resources in bucket policies
An arn:aws:s3vectors: Resource is as dead as an s3vectors action: the
evaluator compares against the generated s3tables ARN, so the rule the
writer believes enforced never matches.
* s3tables: reject s3vectors resources in every ARN partition
* s3tables: test wildcard-partition and mixed-list s3vectors resources
* filer.sync: skip superseded replays before fetching any chunks
On a directory under heavy overwrite, most entries the sync replays are
already superseded on the source. The sink only learned that after chunk
reads failed — or worse, after fetching and uploading bytes a later event
replaces — so the backlog drained slowly and the log filled with 404s.
CreateEntry and UpdateEntry now check sourceMovedOn before touching a
chunk: one entry lookup skips a stale replay entirely. A mid-replication
supersession is still caught by the existing retry gates.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* replication: bound the source liveness probes with a timeout
The supersession checks added a source entry lookup to the replication
path, and every lookup ran on context.Background() with no deadline. A
source that accepts the call but whose metadata store stalls would hold
the sync forever — before a single chunk fetch, even when every chunk is
still readable. Bound the probe at 30s; an unanswered lookup proves
nothing, so replication proceeds and the existing retry gates decide.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* replication: sync the live version a superseded replay points at
Skipping a stale replay assumed a later event always re-carries the
current content. It does not when the superseding write was itself
replicated in from another cluster: those events are dropped before they
reach a filer sink, so the destination could be left missing the file.
When the source reports a live entry newer than the replay, commit that
version through the normal path instead — equivalent to applying the
later event, with no dependence on its delivery. A gone source entry
still skips, since the destination already matches the source.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* replication: reconcile moved entry content after a native rename
AtomicRenameEntry relocates whatever version the destination held. If the
preceding update replay was skipped because the source had already moved
to the new name, the moved entry keeps the old bytes and — with no
further writes — stays stale forever. MoveEntry now runs the same
reconciliation as an in-place update against the rename's snapshot, so
the moved entry picks up pending content changes.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* replication: repair from the live snapshot a superseded replay resolves to
An incremental diff against the replayed oldEntry can resolve manifests the
overwrite already vacuumed, which fails every retry of the event.
---------
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* s3vectors: translate vector bucket policies to the catalog form
The document is stored verbatim on the shared table bucket and evaluated
against s3tables actions and ARNs, so a policy written for the vector API
was a dead rule. Actions now map to the catalog operation the request
authorizes against, and an arn:aws:s3vectors: resource becomes the
bucket/.../table/vector/... ARN the index's catalog entry answers to.
* s3tables: expose PrincipalFor and PolicyContext for façade evaluation
A façade authorizing against its own action namespace needs the same
identity resolution and evaluation context the catalog uses internally.
* s3vectors: keep bucket policies in vector form and evaluate them in the gateway
The earlier approach translated the document to catalog actions and ARNs,
which lost the caller's original policy for GetVectorBucketPolicy and could
not cover data-plane calls that never touch the catalog. The document now
stays in s3vectors form under a dedicated bucket attribute and every
handler evaluates it through authorizeVector against the action and ARN
the request names; a dedicated key also keeps a vector grant from ever
authorizing a catalog call.
* s3vectors: close policy-mutation races and keep vector grants visible
- Authorize Put/DeleteVectorBucketPolicy inside the policy slot's
compare-and-swap, so a grant revoked between reads cannot authorize
the pending write.
- Assert the policy attributes stay absent in the update precondition,
so a concurrent policy or tag write trips the retry instead of being
silently reverted.
- Migrate a raw vector document out of the shared policy slot on
replace or delete, keeping its catalog rules.
- Pass the unresolved identity to PolicyContext so open-by-default
deployments still work.
- Filter ListVectorBuckets on the vector policy itself, so buckets a
caller was granted under s3vectors actions stay listed.
* s3vectors: reuse SnapshotExtended for the vector-policy CAS precondition
* s3vectors: skip buckets that vanish mid-list instead of failing the page
* s3vectors: reject non-vector bucket formats inside authorizeVector
* s3vectors: check bucket format after authorization in authorizeVector
An unauthorized caller probing an ICEBERG bucket got NoSuchBucket,\nleaking that the bucket exists. The format check now runs after the\npermission check so denied callers always see AccessDenied.
* s3vectors: cover legacy shared-slot policy removal on delete
* s3tables: resolve the forwarded identity name like getAccountID in PrincipalFor
A synthesized manager request resolves its caller through the identity-
name branch of getAccountID, which normalizes the name (ARNs collapse to
their trailing segment, whitespace trims) before falling back to the
handler's account ID. PrincipalFor returned the raw name, so ARN-shaped
identities never matched their own grants.
* s3tables: assert the vector policy key in update preconditions
s3vectors-policy is stored on table bucket entries but was missing from
s3tablesExtendedKeys, so a catalog update that read the entry before the
first vector policy was saved could overwrite it without tripping the
compare-and-swap. Asserting the key makes such a race fail instead of
silently erasing the policy.
* s3vectors: reject rules the evaluator cannot honor in vector bucket policies
The shared Statement shape has no negated fields, the vector evaluation
supplies no condition keys, and a document that fails the typed decode
is discarded wholesale at evaluation — including denies the writer
believed active. Reject NotPrincipal, Condition, foreign-service actions
and resource ARNs, and require the typed PolicyDocument decode. Also
cover that an admin-named identity still reaches its own buckets.
* s3vectors: read vector policy fields the way the evaluator decodes them
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* s3tables: match ARN policy principals through caller normalization
Caller principals are normalized ids, so a statement naming a full IAM
ARN could never match: an ARN-scoped deny was silently dead and an
ARN-scoped allow unreachable. Compare arn: specs against the caller
under the same normalizePrincipalID the ownership path uses, while an
ARN caller keeps verbatim wildcard matching.
* s3tables: grant the admin bypass on the caller's actions, not the name
A principal that only normalizes to the admin id skipped every policy
deny. When the request carries an identity, the bypass now requires the
identity's own admin action — the same gate getAccountID applies — while
name-only transports with no action list keep it. PolicyContext takes
the request so it can carry IdentityActions, and the vector context
helpers route through the shared identity keys so the resolved name and
actions agree.
* s3tables: match ARN policy principals only for deny statements
Normalization drops the ARN's account and path, so an allow matched on the
trailing name could grant a same-named caller the statement never verified.
Denies keep the normalized match so the rule cannot be ducked through the
same collapse.
* s3tables: scope service wildcards to their own service
s3tables:* counted as blanket admin inside the shared permission check, so
an identity holding only catalog grants administered s3vectors operations
too. hasAdminActionFor now requires the wildcard to name the operation's
service; the global admin markers and s3:* stay service-agnostic.
---------
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* s3vectors: reuse shared authenticated identity context
* s3vectors: encode empty results as arrays
* test: cover S3 Vectors with Go and Rust volume servers
* ci: require both volume implementations in vector integration tests
* s3vectors: keep identityOf as a shim over the shared identity context
Concurrent branches add identityOf call sites; deleting the helper would
break the build when they merge in either order. Route it through
GetIdentityNameFromContext so every caller resolves the same identity.
* test: wait for the vector engine to register before running
The gateway resolves weed-vector lazily through the master cluster list,
so a listening port does not mean the engine is reachable yet; the first
vector write or query could race KeepConnected and fail. Restore the
cluster.ps registration wait the previous harness had, and ignore the
pytest cache directory the suite creates.
* test: wait for master registration across OSS and enterprise
Use the exact engine registration event after AddClusterNode completes. The current enterprise cluster.ps does not list vector nodes, so waiting on its output prevents the shared fixture from starting there.
* s3vectors: keep contextWithIdentity as a shim alongside identityOf
In-flight callers still write the caller name through it; route the
write through SetIdentityNameInContext so both helpers agree on the
shared key whichever lands first.
An arn:aws:s3vectors: Resource is as dead as an s3vectors action: the
evaluator compares against the generated s3tables ARN, so the rule the
writer believes enforced never matches.
* vector: implement drop_index in the engine
The RPC was a stub; it now resolves the table's vended location and
removes every object under it through the same credential path open uses.
A table the catalog can no longer describe is already dropped, so the call
stays idempotent.
* s3vectors: drop index data after the catalog delete
DeleteTable carries the delete authorization, so it runs first; the engine
DropIndex then removes the data under the index's location. NotFound and
Unimplemented are tolerated so an index whose data went away with the
catalog entry, or an older engine, still answers 200.
* s3vectors: let the catalog delete carry the index data removal
A vector index is colocated: the catalog entry sits at the dataset root,
so DeleteTable's recursive delete already removes entry and data. The
engine DropIndex call afterwards could only ever describe a missing or
recreated record — a no-op in the common case, and a delete of a
replacement index's files when the name was reused in between.
* filer_pb,filer: per-mutation preconditions on ObjectTransaction
A transaction's request-level condition evaluates one entry before any
mutation runs, so a commit guarding two paths — the object it publishes
and the upload directory it assembles — could not be expressed. Give each
ObjectMutation an optional condition (evaluated against condition_key,
else the mutation's target, under the same lock just before the mutation
applies) and report which mutation failed through failed_mutation on the
response, so callers can tell a mutation guard's rejection apart from the
request-level precondition's.
* s3: evaluate multipart completion conditions under the object lock
CompleteMultipartUpload checked conditional headers gateway-side before
committing, and the routed commit transaction guarded only the upload
directory — nothing made If-None-Match: * atomic with the publish, so
concurrent conditional completions of the same absent key could all
report success.
The request's reducible write condition (If-None-Match/If-Match, star or
single strong ETag) now rides the commit as the transaction's request
condition on the owner filer: on the object path for unversioned buckets,
falling back to the .versions directory for suspended and enabled
versioning. The upload-exists guard moves onto the PUT mutation, keeping
its check-before-publish ordering. Conditions the reducer cannot express
stay on the distributed write lock, where check and commit already
serialize.
* filer,s3: resolve the latest version first for versioned conditional writes
* filer: keep directory entries holding content out of the absent-slot rule
* s3: keep a gateway-side upload-exists check for filers without mutation conditions
* filer: honor null-version-is-latest resolving prefer-alt conditions
* drop scratch multipart-race repro programs
* s3: note the write lock does not cover routed writes on non-reducible conditions
* filer: mirror the gateway's remote-etag precedence in condition evaluation
storedEntryETag only knew the extended attribute and the chunk fallback,
so a synced remote entry compared chunk-derived ETags while the gateway
compares the remote ETag. Follow the same precedence: extended attribute,
remote ETag while the entry still holds the remote bytes, Md5, then the
chunk hash — and no ETag for an entry holding none.
* s3: reduce all conditional write headers into owner-lock clauses
Time-based headers, combined conditions, ETag lists, and weak ETags stayed
on the distributed write lock because buildWriteCondition could not express
them, leaving their gateway check racing owner-routed writes. Reduce every
parseable header form to WriteCondition clauses — multiple clauses compose
AND, ETag candidates carry the raw list for the filer's canonicalization,
and RFC 7232 precedence keeps If-Unmodified-Since hidden by If-Match and
If-Modified-Since hidden by If-None-Match — so the owner filer evaluates
them under the same object-path lock as the commit. Only unparseable
headers still fall back, where the lock path reports the same parse error.
* filer_pb,filer: per-mutation preconditions on ObjectTransaction
A transaction's request-level condition evaluates one entry before any
mutation runs, so a commit guarding two paths — the object it publishes
and the upload directory it assembles — could not be expressed. Give each
ObjectMutation an optional condition (evaluated against condition_key,
else the mutation's target, under the same lock just before the mutation
applies) and report which mutation failed through failed_mutation on the
response, so callers can tell a mutation guard's rejection apart from the
request-level precondition's.
* s3: evaluate multipart completion conditions under the object lock
CompleteMultipartUpload checked conditional headers gateway-side before
committing, and the routed commit transaction guarded only the upload
directory — nothing made If-None-Match: * atomic with the publish, so
concurrent conditional completions of the same absent key could all
report success.
The request's reducible write condition (If-None-Match/If-Match, star or
single strong ETag) now rides the commit as the transaction's request
condition on the owner filer: on the object path for unversioned buckets,
falling back to the .versions directory for suspended and enabled
versioning. The upload-exists guard moves onto the PUT mutation, keeping
its check-before-publish ordering. Conditions the reducer cannot express
stay on the distributed write lock, where check and commit already
serialize.
* filer,s3: resolve the latest version first for versioned conditional writes
* filer: keep directory entries holding content out of the absent-slot rule
* s3: keep a gateway-side upload-exists check for filers without mutation conditions
* filer: honor null-version-is-latest resolving prefer-alt conditions
* drop scratch multipart-race repro programs
* s3: note the write lock does not cover routed writes on non-reducible conditions
* mount: keep the shared directory build out of the requester's context
The shared directory build behind EnsureVisited is a singleflight call: the
first requester for a path owns the build, and every later requester for the
same path waits on that same build. The build ran on the owner's request
context, so a FUSE request that returned -- or that the kernel interrupted --
canceled the build for everybody waiting on it.
A waiter whose own context was still alive then received somebody else's
cancellation. weed/mount/weedfs_dir_read.go classified a canceled read as
EINTR only when the local context was done, so such a waiter fell through to
fuse.EIO: the kernel handed EIO to readdirent, kubelet's applyFSGroup failed,
and the volume never mounted.
Build the shared directory on a context that keeps the request's values but
not its cancellation, and report a canceled read as EINTR even when the local
context is still alive. Waiters still abandon on their own context, so a
canceled request stops waiting immediately.
This follows up #11717, which made the wait abandonable and mapped the local
cancellation to EINTR, but left the shared build itself on the owner's
context. The surrounding build lifecycle calls already use
context.Background() for the same reason.
Test: TestEnsureVisitedSharedBuildSurvivesOwnerRequestCancel reproduces the
cancellation with a fake filer and fails without this change.
* mount: classify gRPC-canceled directory listings as EINTR
* mount: log only real directory read failures, not interruptions
* mount: keep shared section refresh out of the requester's context
* mount: let section-refresh waiters leave on their own context
The shared refresh flight now outlives the requester that started it,
but callers still blocked in singleflight.Do until it finished — an
interrupted directory read could not stop waiting, and the caller's
own deadline never applied to the wait. DoChan + select on the
caller's context restores early abandon while the detached refresh
still completes for the remaining waiters.
* ci: re-run FUSE integration after a flaky writeback-cache failure
* mount: refresh a stale entry once before reporting a read error
A read resolves chunk views from the handle's entry. A concurrent
overwrite commits a new entry whose replaced file ids the filer queues
for deletion; the open handle only learns the new chunk list when the
metadata invalidation lands. A fetch that lands between the volume
delete and the invalidation names data that is already gone and the
read reports EIO. On a chunk read error, wait out any in-flight async
flush, re-read the entry from the filer, install it when it outranks
the handle, and retry once. An unchanged version means the failure is
genuine.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* mount: fence the read-error entry refresh by file identity
Two guards on refreshEntryForRead's install path: the unlocked
dirtyMetadata read raced with Write under the handle lock — drop it,
installAckedEntry already checks under that lock, and it now fences
isDeleted handles the same way. And a saved path can name a different
file after unlink/reuse, so refuse the refresh when the fetched inode
disagrees with the one the handle holds.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: user.email <chrislusf@users.noreply.github.com>
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* s3tables: carry encryptionConfiguration in table bucket metadata
* s3vectors: accept and return encryptionConfiguration on vector buckets
The configuration is validated (sseType AES256 or aws:kms) and stored on
the bucket record, so the API round-trips what the caller declares.
* s3vectors: reject caller-chosen encryption keys on vector buckets
* s3tables: reject caller-chosen encryption keys on table buckets
* s3tables: reject unknown sseType in bucket encryptionConfiguration
The bucket check rejected aws:kms and kmsKeyArn but stored any other
sseType — a made-up or mistyped value would round-trip through
GetTableBucket as an encryption claim the storage does not keep. The
validator now admits only the AES256 declaration, shared so the vector
layer can drop its copy.
* s3vectors: delegate encryption validation to the shared catalog check
* s3tables: accept the sseAlgorithm name in bucket encryptionConfiguration
S3 Tables declares the algorithm as sseAlgorithm while S3 Vectors uses
sseType; the shared check now accepts either spelling, still admits only
AES256, and still refuses any caller-chosen key.
* s3vectors: call the shared encryption validator directly
* s3vectors: require each API's own encryption field name
The shared validator accepted sseAlgorithm on the vector path, so a
CreateVectorBucket carrying the s3tables field stored a configuration
GetVectorBucket could never echo back in the vectors shape (and vice
versa). ValidateEncryptionConfiguration now takes the wire name its
caller expects; the vectors create path normalizes to sseAlgorithm for
the catalog and GetVectorBucket renames it back to sseType.
* s3vectors: bound QueryVectors topK at the AWS limit of 30
topK passed through unbounded; AWS caps a query at 30 results, so a
larger request is a ValidationException before it reaches the engine.
* s3vectors: raise QueryVectors topK bound to the AWS limit of 10,000
* s3vectors: validate QueryVectors topK before the catalog lookup
* s3vectors: cover the QueryVectors topK bounds
* util: do not bind outbound dials to a loopback -ip.bind
`weed server -ip.bind=127.0.0.1` keeps every listener local, but the
same address became the source of outbound connections, so the filer
dialled its remote mount from 127.0.0.1 and failed with "connect:
invalid argument". A loopback source cannot reach off the host; leave
the source to the OS, as for a wildcard bind.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3vectors: put, get and delete vector bucket policies
The AWS S3 Vectors policy operations had no handlers; the policy document
is stored on the shared table bucket, so the handlers delegate to the
catalog's PutTableBucketPolicy, GetTableBucketPolicy and
DeleteTableBucketPolicy and keep its guard.
* s3vectors: treat ICEBERG buckets as absent in policy handlers
GetVectorBucketPolicy/PutVectorBucketPolicy/DeleteVectorBucketPolicy
delegated straight to the table bucket, so a bucket created for ICEBERG
tables — invisible to every other vectors operation — could still have
its policy read and written through the vectors API. Gate all three on
lanceBucket, the same format check DeleteVectorBucket and CreateIndex
already apply.
* test/vectors: cover vector bucket policy put/get/delete
Round-trips a policy document through the new ops and asserts a get on
the deleted policy surfaces NotFoundException, so the boto3 conformance
run exercises the added handlers end to end.
* s3vectors: validate the bucket ARN and policy document
Bucket-level operations now reject an index ARN and an ARN minted for
another service, and PutVectorBucketPolicy refuses a document carrying
s3vectors actions — the evaluator only honors s3tables actions, so such a
rule would be a dead grant.
* s3vectors: validate the bucket ARN on the remaining bucket-level operations
* s3vectors: cover bucket ARN and policy document validation
TestBucketARNs pins the accepted forms and the index/foreign-service
rejections; TestValidatePolicyDocument covers a valid document, a
malformed one, and the dead s3vectors action namespace.
* s3vectors test: use an enforceable action in the bucket policy test
The catalog refuses s3vectors actions in a stored policy, so the put call
rejected the document before the get and delete assertions ran.
---------
Co-authored-by: Ólafur Bjarki Bogason <olafur@genkiinstruments.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* s3vectors: cap PutVectors at 500 items and check data against the index dimension
A batch could carry any number of vectors and a vector of the wrong
dimensionality reached the engine: AWS caps a PutVectors call at 500 items
and requires each vector to match the index's declared dimension.
* s3vectors: cover the PutVectors batch and dimension checks
* s3vectors: reject query vectors that do not match the index dimension
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* s3vectors: cover the PutVectors batch size bounds
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* s3vectors: cover the catalog-backed PutVectors handler path
Run PutVectors through the real handlers over the in-memory catalog: a
matching vector reaches the engine, a mismatching one is rejected with
ValidationException before PutVectors is called, and a full
vectorsMaxPutBatch batch is accepted.
---------
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* iceberg maintenance: an unreadable parquet file fails the job, not the process
parquet.NewReader panics on any file it cannot open (wrong magic, a
truncated footer, zero bytes), and parquet-go's row decoder panics on a
corrupt page under a valid footer. Maintenance runs in the worker's
goroutines with no recover, so one bad delete or data file in any table
ended the whole process.
- Every parquet open in compaction goes through newParquetReader, which
opens with parquet.OpenFile and returns the error.
- A panic while decoding a delete file or a data file's rows is recovered
into an error naming the file (recoverRead/guardRead).
- As a backstop, detection recovers per table and Execute per operation:
the table is skipped or the operation fails, and the worker keeps
running.
A failed delete file still fails the table's compaction (no commit), as a
read error did before; a failed data file still skips only its bin.
* iceberg maintenance: cache a remote-only file before reading it
On a bucket mounted with remote.mount, a file that is not cached locally
has no chunks and no inline content, only its RemoteEntry. Maintenance
read it as an empty file: a parquet reader failed on zero bytes (before
the previous change, it panicked and ended the process) and a manifest
failed to parse, so a table whose files had been uncached could not be
maintained.
lookupFileByIcebergPath, the lookup every read goes through, now calls
CacheRemoteObjectToLocalCluster for a remote-only entry, as the S3
gateway does on a read, and reads the entry it returns; the call is tried
twice, and an entry still remote-only after caching is an error. The
lookups that only resolve a path (orphan removal) do not cache.
readEntryContent reports a file with no content and no chunks as empty
instead of returning zero bytes.
The fake volume server in the tests now serves chunk reads, so the
remote-only tests read the cached file through the chunk reader as a
real one is read.
* iceberg maintenance: bound the metadata location read before caching a remote file
* iceberg maintenance: bound the metadata read on the entry the cache returns
* iceberg maintenance: enforce the metadata size bound inside the remote cache call
The worker checks the resolved entry before asking the filer to cache a
remote-only object, but the filer resolves the entry again at cache time
and downloads whatever its remote size then says. A replacement landing in
between could push an oversized object through the download even though
the post-cache check would refuse to read it.
CacheRemoteObjectToLocalClusterRequest gains an optional max_size; the
filer rejects an over-limit remote object before the download starts.
loadMetadata passes metadataFileMaxBytes; data-file reads leave it unset.
* iceberg maintenance: key the remote cache singleflight by the size limit
Callers sharing one flight share its outcome, including a bounded
caller's over-limit rejection: an unrestricted reader joining the same
path would fail even though its own bound allows the download. The
limit joins the key so only callers with the same bound share a flight.
* iceberg: gofmt compact_order_test.go
* filer: bound remote cache callers before the shared download
Each caller's max_size is now enforced on the entry the filer resolves
before that caller joins the singleflight, which is keyed by path again.
A bounded caller's rejection is never shared with other callers, and
callers with different bounds share one download instead of pulling the
same remote object twice.
* filer: bind the remote cache size bound to the downloaded entry
The pre-flight check and the shared download resolve the entry
separately, so a remote metadata refresh landing in between lets a
larger object through: the download ran to the refreshed size and only
the reader's post-check refused it. The bound is now re-checked inside
the flight on the entry the download streams.
The rejection is typed so a co-tenant sharing the flight does not
inherit it: callers whose own bound permits the reported size retry on
a fresh flight instead of failing for another caller's limit.
---------
Co-authored-by: user.email <chrislusf@users.noreply.github.com>
* mount, s3: clarify allowOthers requirement and bucket name error
`weed mount -allowOthers` defaults to true. On Linux, a non-root
user's mount then needs `user_allow_other` in /etc/fuse.conf, but the
help text only said "allows other users to access the file system".
Say so in the help, and point at -allowOthers=false.
Also fix the grammar of the bucket name length error ("must between
[3, 63]" -> "must be between 3 and 63").
Help and error text only; no behaviour change.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3: normalize the remaining VerifyS3BucketName error messages
The length message was clarified; the rest of the function still had
inconsistent phrasing ("name must start", "cannot be ip addresses",
"prefix xn-- ... in bucket prefix"). Align them on "bucket name ..."
wording matching the AWS bucket naming rules.
---------
Co-authored-by: pingqiu <pingqiu999@gmail.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: user.email <chrislusf@users.noreply.github.com>
`weed server -ip.bind=127.0.0.1` keeps every listener local, but the
same address became the source of outbound connections, so the filer
dialled its remote mount from 127.0.0.1 and failed with "connect:
invalid argument". A loopback source cannot reach off the host; leave
the source to the OS, as for a wildcard bind.
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
The vectors gateway backs off discovery for 5s between resolutions, but
the backoff also suppressed rediscovery after an empty lookup. A worker
that landed its KeepConnected registration a moment after the first
query stayed invisible for the whole window — longer than a client's
retry budget — so the S3 Vectors e2e flaked: CreateIndex raced the
registration, got InternalFailure, and every boto retry fell inside the
same 5s fast-fail window.
With no cached backend, recheck every 500ms instead; the 5s interval now
only bounds membership refreshes against a live client, and an absent
worker still cannot flood the master with lookups. The empty-membership
error also becomes codes.Unavailable so the API answers 503 rather than
InternalFailure.
cluster.ps lists registered vector nodes like the other node types, and
the e2e harness polls it for the engine's registration instead of only
checking the gRPC port — a listening engine is not discoverable until
the registration lands.
* volume: remount ec shards the rebuild replaced under a mounted handle
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* ec.rebuild: keep copied shards the rebuild regenerated in place
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* volume: reopen regenerated ec shards in place instead of remounting
UnmountEcShards and MountEcShards send deltas through separate buffered
channels, so a remount can reach the master as add-then-delete and drop
the repaired shard from lookups until the next full EC heartbeat.
ReopenEcShard swaps the mounted shard's descriptor under the volume lock
instead: registration, disk type, and heartbeat state never flip, and
readers holding the old shard degrade to a clean error the parity path
already recovers from.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* rust volume: reopen rebuilt ec shards in place, mirroring Go
volume_ec_shards_rebuild recreates a shard file under a mounted handle
whose descriptor still serves the detached inode (a mounted shard whose
file was deleted under it). DiskLocation::reopen_ec_shard swaps the
mounted descriptor for a fresh open, matching Go's ReopenEcShard — the
registration and heartbeat view never flip.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* volume: fail ec rebuild when a regenerated shard cannot be reopened
* rust volume: fail ec shards rebuild when a rebuilt shard cannot be reopened
* volume: repoint mounted ec shards after rebuild across disks
A mounted shard whose file was deleted under the mount is "missing" to
the rebuild: the regenerated file lands on the rebuild location while
the handle keeps serving the detached inode on another disk. Reopening
the mounted disk then failed the whole RPC, so the caller never saw the
rebuilt ids and its deferred cleanup deleted the regenerated copies.
Repoint every mounted handle of the rebuilt volume instead of only this
run's outputs: reopen in place when the rebuilt file is on the mounted
disk, otherwise drop the stale handle and remount the shard from
whichever local disk holds the file via a delta-free load split out of
MountEcShards — a same-server move, so the master's registration never
flips. A shard that can be neither reopened nor remounted is left
unmounted rather than serving replaced bytes, and the repoint step no
longer fails the RPC, so RebuiltShardIds always reaches the caller.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* rust volume: repoint mounted ec shards after rebuild across disks
Mirror the Go repoint loop: iterate every mounted shard of the rebuilt
volume, reopen in place when the rebuilt file is on the mounted disk,
and otherwise drop the stale handle and remount the shard from
whichever local disk holds it. The repoint no longer fails the RPC, so
rebuilt_shard_ids still reaches the caller and its copy cleanup keeps
the regenerated shards.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* rust volume: still rebuild ecx and repoint mounts when no shard is missing
* master: keep a node in an ec shard's lookup while a disk still holds it
A rebuild that relocates a mounted shard between a node's disks reports
the move as a delete on the source disk plus an add on the destination.
Registering first is a no-op because shard lookups key on the node, then
the unregister dropped the node outright — and steady-state heartbeats
never restore it, so readers and later rebuilds lose the repaired copy.
Skip the delete while another disk on the node still holds the shard.
UnregisterDataNode now clears the node's own ec records before dropping
its registrations, so the retained-shard check cannot skip the teardown,
and each unregister gets a ShardsInfo copy since DeleteEcShard drains
the shared one.
* rust volume: fold the journal on a mounted index instead of rewriting it
The rebuild handler regenerated .ecx through File::create even while the
volume was mounted, truncating the index under readers that search the
open descriptor with its old size — a live lookup could hit EOF mid-scan.
Go folds the deletion journal in place on a mounted volume; do the same
through rebuild_ecx_from_journal and reserve the full regen for the
unmounted path, where nothing holds the file open to interrupt.
* volume, rust volume: report rebuilt shards on late failures, fix rust corrupt discovery
Two review findings the repoint rework left open:
- A RebuildEcxFile failure still returns a plain error after shards were
regenerated, so the caller discards the response holding the rebuilt
ids and the deferred cleanup deletes copies the rebuild regenerated
in place. The Go handler now packs the ids into the gRPC status
details and the shell pulls them back out on error; the Rust handler
carries them in grpc-status-details-bin the same way.
- The Rust rebuild still counted a sidecar-corrupt shard as present, so
it fed bad bytes into reconstruction and never regenerated the shard.
Discovery now loads the generation-0 .ecsum the way Go does, verifies
every present shard, and reclassifies mismatches (and zero-size
residue) as rebuild targets written to a .rebuilding temp file,
verified, and atomically renamed over the original only after every
output passes the sidecar check. Honors unsafe_ignore_sidecar, added
to VolumeEcShardsRebuildRequest to match the Go proto.
* volume: repoint sibling journals sharing a folded .ecj after rebuild
RebuildEcxFile unlinks the deletion journal and the mounted path only
reopens its own handle — a sibling mount journaling into the same file
keeps appending to the detached inode, and those deletes never reach a
path-based reader again. Reopen every mounted volume sharing the index
base before leaving the rebuild.
* rust volume: repoint sibling journals sharing a folded .ecj after rebuild
rebuild_ecx_from_journal replaces the journal and reopens only its own
handle; mounts on sibling disks sharing that .ecj keep the detached
handle, so their next appends are lost at restart. Adopt the folded
journal on every mounted runtime sharing the path before releasing the
store lock, clearing their in-memory set — every journaled id is now an
.ecx tombstone.
* ec: swap the deletion journal handle only after the new file opens
A failed reopen left ecjFile nil, so every later delete returned
"ec volume closed". Open first, then close and swap — mirroring the
Rust adopt_folded_journal ordering.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* ec.rebuild: fail the rebuild when a sibling journal cannot be repointed
A warning left the sibling mount journaling to a detached inode while
the RPC reported success. Propagate the error like the Rust handler's
adopt_folded_journal map_err.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* ec: keep deletes loud when the journal reopen cannot open the replacement
Reverts the swap-on-success ordering: a detached handle kept on failure
lets deletes append and sync to the removed inode, then vanish at
restart. Closing first also frees the descriptor before the open.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* rust volume: drop the folded journal handle before reopening it
adopt_folded_journal and the post-fold recreate now clear ecj_file
before opening the replacement, so a failed reopen leaves deletes
erroring instead of appending to the detached inode. Mirrors Go's
ReopenDeletionJournal.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* ec: drop the cached journal size when its reopen fails
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* remote: reuse one guarded remote storage client per configuration
BuildGuardedRemoteStorageClient (the filer's lazy remote stat and list,
remote deletes and renames, the volume server's cold reads, the S3
gateway's remote reads) built a new client, with its own http.Transport,
on every call. Each transport kept its connection idle for
IdleConnTimeout (60s) and was never reused or closed, so a steady stream
of remote lookups held one open connection per request. An endpoint
that caps connections per source then refuses every new connection.
- remote_storage: GetRemoteStorage's per-name cache becomes a bounded
LRU (WEED_REMOTE_CLIENT_CACHE_MAX, default 64) keyed by kind, name, a
hash of the whole configuration and a credentials version, built
outside the lock once per key (singleflight) from a copy of the
configuration. A client leaving the cache has its idle connections
closed. GetRemoteStorageWith exposes the cache to other builders.
- BuildGuardedRemoteStorageClient caches the guarded client per
configuration. The endpoint is checked against the deny-list when a
client is built; every new connection still dials through the guarded
dialer. A gcs credentials file, and for an S3-compatible remote using
the default credential chain the shared credentials file and AWS_*
variables, are part of the key by content.
- The shared transport keeps 32 idle connections per host, caps
connections per host (WEED_REMOTE_MAX_CONNS_PER_HOST, default 64) and
has dial, TLS handshake and response header deadlines, so a hung
remote frees its slots.
- The filer drops the clients of removed remotes (and all but the newest
client per name) on every /etc/remote reload, also when /etc/remote
does not exist; reloads are serialized.
* remote: drop clients whose build spans a reload or whose remote is gone
* server: hash the shared AWS config file into the guarded credential version
* server: bound a stalled body on the shared remote transport
* remote: waive path-injection on the configured credentials file
* remote: keep caching remotes the local filer does not name
* remote: drop the inline codeql waiver the analyzer ignores
* remote: version the plain gcs client's credentials file
* remote: hash the environment's inline gcs credentials
GOOGLE_APPLICATION_CREDENTIALS can hold the JSON itself, which the gcs
client accepts; it is not in the configuration hash, so a changed value
kept the old client.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* remote: version the well-known gcs credentials file
With neither the configuration nor GOOGLE_APPLICATION_CREDENTIALS set,
the SDK reads application_default_credentials.json; rotating it left the
cache key unchanged and the old client behind.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* remote: share the gcs credentials version with the guarded client
The guarded gcs client resolves the same environment and well-known
credentials as the plain one; keying it only on a configured path left
those rotations invisible to the cache.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* remote: version s3 keys resolved from the environment
A remote whose access and secret keys both arrive through its type's
environment fallback (aliyun, baidu, contabo, filebase, storj, tencent)
never entered the default-chain branch of guardedCredentialsVersion, so
an in-process environment change kept serving the stale cached client.
The version now hashes the resolved keys for every S3-compatible remote.
* volume: reuse one remote storage client per configuration (rust)
fetch_and_write_needle built a fresh aws_sdk_s3::Client per request,
so every cold read kept its connector's idle connections without ever
reusing them — the same leak the Go side fixes. Cache clients keyed by
the configuration's protobuf encoding in an LRU of
WEED_REMOTE_CLIENT_CACHE_MAX entries (default 64), mirroring
weed/remote_storage/client_cache.go. An evicted entry drops the client's
last Arc and with it the connector's idle connections. The per-request
endpoint validation is unchanged: Rust has no dial-time guard yet, so
the up-front DNS check stays on every call.
---------
Co-authored-by: user.email <chrislusf@users.noreply.github.com>
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* mount: rebuild dropped directory listings on lookup
Invalidation, idle eviction, and a kernel Forget all clear a directory's
cached-listing flag, but only an expired listing was rebuilt on lookup — a
dropped one read through per entry until a ReadDir happened to run. Track
directories that once held a listing in wasCachedDirs, which survives those
drops, and let ShouldRebuildExpiredDir rebuild them on the next lookup.
Never-cached and read-through directories keep reading through.
* mount: re-validate a stale directory section on lookup
A burst of remote changes invalidates only the directory section it lands
in, but nothing re-validates that section until a ReadDir runs — so a
lookup-heavy workload reads through per entry indefinitely. EnsureNameFresh
re-lists just the section covering the looked-up name, after which lookups
in it are served from the cache again.
* mount: wait for pending flush before a stale-section refresh on lookup
A section refresh lists the range from the filer; without the wait a
pending async flush of the looked-up entry could land after the listing
and leave the refreshed section holding pre-flush metadata — the same
hazard the rebuild path already waits out.
* mount: bound the dropped-listing rebuild history
wasCachedDirs grew one full path per directory ever cached, removed only
by the read-through mark, so deleted and forgotten directories
accumulated forever. Remove the record when its name is deleted, move it
on rename so rebuild eligibility travels with the directory, and cap the
history with FIFO eviction — past the bound an evicted directory reads
through per entry, the same end state as a never-cached one.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* mount: drop oversized stale sections and back off failed refreshes
A section that outgrew one refresh failed EnsureNameFresh with
ErrRefreshRangeTooLarge on every lookup, repeating the multi-page
listing before each per-entry read, and other refresh failures retried
with no cooldown. Drop the cached listing on ErrRefreshRangeTooLarge so
a later lookup rebuilds it re-tiled, as ReadDir does, and stamp the
rebuild attempt on other failures so retries back off under the same
cooldown the rebuild gate uses.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* mount: version hard-link sibling syncs at the committed log position
* mount: tag dropped-listing queue slots so stale ones cannot evict
RemovePath, MovePath, and MarkDirectoryReadThrough delete the rebuild
record but leave the FIFO slot behind, so a path removed and re-cached
holds two queue entries. When the stale slot reaches the front it evicts
the newer record, and the directory reads through per lookup until the
next listing. Tag each record with an insertion generation and only
evict on a match, keeping the newer record for its own slot's turn.
* mount: refresh hard-link siblings from the filer, not the cached copy
Stamping a cached sibling with the mutation's version claimed that
position for content that may predate it — an in-flight event with
newer chunks or attributes then hits the version gate and the sibling
serves stale data until a refresh. Fetch each cached sibling's entry
from the filer after the commit and record the higher of the fetched
and committed positions, falling back to the in-place counter patch
only when the fetch itself fails.
* mount: back off failed section refreshes per section, not per directory
A failed refresh of one stale section marked a directory-wide attempt,
so every other stale section in that directory waited out the cooldown
with per-entry filer reads. Key the attempt timestamp by the section's
lower bound so unrelated ranges still refresh on their own schedule.
* mount: fence hard-linked entries by the group's newest write position
Every path of a hard-link group reads one shared content blob, but the
version gate only tracks per-path records — a sibling absent from
inodeToPath got no version from the sibling sync, so a stale section
refresh or event could roll the shared blob back for all of them.
Record a monotonic version floor per hard-link id wherever versioned
content lands, and check it in the section-refresh and event gates.
* mount: keep hard-link names landing when the group floor fences content
The per-group version floor rejected a delayed event's newEntry while
its oldPath still applied — a rename could drop the source without
adding the destination, and a section refresh could skip then mark a
live link fresh. Rebase the group's current cached content onto the
new path so the name lands without rolling the shared blob back, only
skip refresh entries already cached, and gate the sibling refresh
write under the same lock so a racing event or delete wins.
* mount: rebase hard-link entries from the shared blob when no sibling is cached
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* mount: park group content on uncached hard-link names a stale refresh installs
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* mount: stale the destination section when a fenced hard-link name cannot land
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* mount: bound the sibling hard-link refresh under one request deadline
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* mount: replace a reused-name occupant the hard-link floor would skip
* mount: claim no version for a sibling counter patched without a fetch
* mount: vacate a reused-name occupant without decrementing its group
A sibling write can fold the occupant's filer-side unlink into the shared
blob counter while the stale entry is still cached, so replacing it through
the plain insert may decrement a group that no longer counts the name and
drop a blob live siblings still read. Vacate the path like a rename first;
the counter then only ever drifts high, which leaks a blob rather than
losing shared content.
* mount: keep a fetch-failed sibling's version when patching its counter
The fallback marked the entry unversioned, which fences nothing — a
replayed delete older than the cached content could remove a live link.
Claim the version the entry already holds instead: newer in-flight events
still land and correct it, older replays stay fenced.
---------
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* admin: warn on dashboard when cluster nodes run mixed versions
A partially-finished upgrade leaves servers on different releases, and
nothing surfaces that drift today. The dashboard now compares the
binary version reported by every node:
- volume servers are asked directly via VolumeServerStatus (the
heartbeat topology carries no version), fanned out with bounded
concurrency and a 60s cache since versions only change on restart
- filers, S3 servers, and message brokers already report their version
over KeepConnected; ListClusterNodes returns it, the dashboard just
never read the field
- masters are excluded: raft membership does not carry versions
When every node matches, the Cluster panel shows the shared version.
When they differ, a warning banner calls out the mixed cluster and the
panel lists each node type with its count grouped by version, e.g.
"31x 4.39, 24x 4.44". Version columns on the volume server, filer, and
S3 tables (and the dedicated volume servers page) identify which nodes
are stale. Versions are compared without the volume-size-limit prefix
("8000GB 4.44 abc123" -> "4.44 abc123") so capacity config does not
masquerade as version drift; nodes reporting nothing count as
"unknown".
* admin: run the volume version queries concurrently, not under the map lock
The per-server VolumeServerStatus call ran while holding the versions
mutex, so the semaphore-bounded fan-out was fully serialized: every
server's RPC waited for the previous one, up to the 3s timeout each.
* admin: export the volume server table's real columns
The new Version cell shifted every later column, so the CSV read the
version as Rack and dropped Max Volumes, EC Shards and Usage a slot
early. Map the cells the table actually renders and add Version to the
export.
* admin: warn on mixed versions only when nodes report versions
An empty or master-only cluster produced a non-uniform summary with an
empty node list, which rendered the "Mixed cluster versions detected"
banner and warning icon over nothing. Mixed() requires reported nodes,
and the version card shows a dash when none answered.
* iceberg maintenance: bound decimal, time, uuid, fixed and rescaled timestamp columns on compacted files
The column statistics written for a compacted data file give lower and upper
bounds only for booleans, integers, floats, dates, strings, binaries and
timestamps already stored in the Iceberg type's unit. Every other column is
left unbounded, so a reader can never skip a compacted file on it: a
DECIMAL(18,2) column written by DuckDB (stored as INT64) loses its bounds
after compaction, as does a millisecond timestamp, a TIME or a UUID.
boundLiteral now also converts:
- DECIMAL stored as INT32 or INT64 at the column's scale. A decimal stored
as a byte array still gets no bound: Parquet compares those statistics as
bytes, which does not order negative two's-complement values;
- TIME, TIMESTAMP, TIMESTAMPTZ and TIMESTAMP_NS from any Parquet time unit,
rescaled to the Iceberg unit; a value rescaled to a coarser unit is
rounded down for a lower bound and up for an upper bound so it stays a
bound, and a value that would overflow gets none;
- UUID and FIXED[n] from a FIXED_LEN_BYTE_ARRAY of the right length;
- a DOUBLE column read from a FLOAT file.
arrow-go becomes a direct dependency (decimal128, already in the module
graph through iceberg-go).
* iceberg maintenance: bound TIME(MILLIS) columns stored as INT32
* iceberg maintenance: bound timestamptz_ns columns too
---------
Co-authored-by: user.email <chrislusf@users.noreply.github.com>
* iceberg maintenance: an unreadable parquet file fails the job, not the process
parquet.NewReader panics on any file it cannot open (wrong magic, a
truncated footer, zero bytes), and parquet-go's row decoder panics on a
corrupt page under a valid footer. Maintenance runs in the worker's
goroutines with no recover, so one bad delete or data file in any table
ended the whole process.
- Every parquet open in compaction goes through newParquetReader, which
opens with parquet.OpenFile and returns the error.
- A panic while decoding a delete file or a data file's rows is recovered
into an error naming the file (recoverRead/guardRead).
- As a backstop, detection recovers per table and Execute per operation:
the table is skipped or the operation fails, and the worker keeps
running.
A failed delete file still fails the table's compaction (no commit), as a
read error did before; a failed data file still skips only its bin.
* iceberg maintenance: test a corrupt page inside a merge fails it naming the file
* iceberg maintenance: assert a skipped bin's files stay in the snapshot
---------
Co-authored-by: user.email <chrislusf@users.noreply.github.com>
* iceberg maintenance: apply equality deletes only to older data files in their partition
Compaction applied every equality-delete key to every data file of every
bin. The Iceberg spec ("Scan Planning") applies an equality delete file only
to data files with a strictly lower data sequence number, and only to data
files in the delete file's own partition unless its spec is unpartitioned.
An upsert writer (Flink, the Iceberg Kafka Connect sink, CDC tools) commits
an update as the row's new version plus an equality delete of its key in the
same commit, so the compaction deleted both the old and the new version of
every updated row.
- equalityDeleteGroup is keyed by equality field ids and scope (unpartitioned,
or spec id plus partition) and keeps, per key, the highest data sequence
number of the delete files carrying it. A delete file without a data
sequence number is an error rather than a guess.
- Partition values are compared typed through the spec's partition type, so
one partition decoded in different Go shapes (a uuid as uuid.UUID or
[16]byte, a long as int32 or int64, a null present or missing) is one
partition. A spec whose fields are all void is unpartitioned. A delete whose
scope cannot be decided (unknown spec, no partition values under a
partitioned spec, a value that does not convert) stops the compaction; a bin
whose own partition cannot be typed while a delete of its spec exists is
not compacted.
- Each bin is merged with only the groups that apply to its partition, and a
row is deleted only by a key whose sequence number is higher than its data
file's.
- A where filter no longer drops unpartitioned delete files (the clause's
field is not in their spec), which let deleted rows return in a filtered
compaction.
Tests: the table fixture now commits its delete files after the data (data
sequence number 1, deletes 2), as writers do; hand-built entries in the unit
tests get sequence numbers. New eqdelete_sequence_test.go covers rows written
with and after their delete, partition scope, unpartitioned and all-void
specs, typed partition matching, missing sequence numbers and partition
values, where filters, and group-key collisions.
* iceberg maintenance: split the undecidable-scope branch in deleteInPredicateScope
---------
Co-authored-by: user.email <chrislusf@users.noreply.github.com>
* s3: accept bare composite checksum on CompleteMultipartUpload
Clients such as minio-go send the composite object checksum without the
"-<parts>" suffix. Compare against the base64 part of the computed
composite value as well as the full form.
* s3: test object checksum matching with and without part suffix
* filer.redis3: prefix every key via getKey, honoring keyPrefix
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* filer.redis3: read username and keyPrefix options into client config
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* filer.redis3: reject a username configured without a password
go-redis only sends AUTH when a password is set, so a configured username
with an empty password silently connects as the default user instead of
the intended ACL user. Fail loudly at initialize instead.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* filer.redis3: cover keyPrefix scoping in the store tests
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* filer.redis3: derive the test prefix and kv key from the per-run directory
---------
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* filer.remote.sync: skip an event the filer has written past, not only rewritten
isSuperseded only looked for chunks the event named that the filer no
longer holds. A file copied in through a mount logs a create while the
copy is still running: the mount flushes metadata with chunk uploads in
flight, so that chunk list can have holes. ChunkStreamReader stops at
the first hole, the multipart upload sends a short part ("ContentLength=
8388608 with Body length 4194304"), and since every chunk the event
names is still live the event was never superseded. The daemon retried
it forever and the offset never moved past it.
Treat chunks the filer has added since the event the same way. The
event that added them follows in the log and uploads the finished file.
* filer.remote.sync: skip a holed snapshot even while the filer still holds it
A mid-write snapshot the writer never finished is neither deleted nor
written past: the filer holds it unchanged, so the superseded check never
fires and every retry reads short at the same gap, pinning the sync offset
forever. The upload can never succeed for that entry — skip it like a
superseded one. A finishing write logs its own event and uploads the full
content; a dead write has nothing readable to lose.
The check also covers a chunkless entry that is not a dropped local copy
(e.g. a truncated remote-only entry): its reader yields zero bytes against
a nonzero size, which can silently clobber the remote object with an empty
body where the backend does not verify the length.
* filer.remote.sync: upload a holed entry zero-filled instead of skipping it
Skipping the event left a completed sparse file absent or stale on the
remote, and the coverage check on the raw chunk list missed both gaps a
manifest's bounding extent hides and a FileSize smaller than the chunk
extent. ChunkReadAt already resolves manifests the way the upload reader
does and zero-fills every uncovered range up to the effective size, so
route the upload through it: a sparse file, a mid-write snapshot, and a
chunkless entry all send the bytes a read of the entry returns, and a
hole can neither wedge the event nor publish a truncated object.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* filer.remote.sync: skip a superseded entry on reader-construction error too
---------
Co-authored-by: Alex Russell <alex@sengine.cloud>
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* s3: return x-amz-copy-source-version-id from CopyObject and UploadPartCopy
* s3: cover x-amz-copy-source-version-id on latest copies and part copies
The explicit-version case was already covered; add a .versions pointer
fixture so the unversioned request resolves the latest version and
asserts the header on CopyObject, plus the same assertion on
UploadPartCopy.
* s3: carry the resolved id on a version file that lacks it
A .versions child written before the version-id attribute resolves to an
empty source version even though the pointer or the rescan already knows
it. Stamp the id on the returned entry so copy and get readers report it;
the version copy tests now drop the attribute to cover that shape.
* s3: report the null version in x-amz-copy-source-version-id
A regular-path source in a versioning-configured bucket is the null
version; echo it like every other version id, matching the GET path and
AWS. Unversioned sources still report no header.
* s3: report the version a self-copy retry actually read
---------
Co-authored-by: user.email <chrislusf@users.noreply.github.com>
* s3: allow PutBucketAcl when auth is off
* s3: resolve the PutBucketAcl owner against the recorded owner
The unauthenticated path now resolves the account as anonymous, but XML
ACLs were still checked against it while GetBucketAcl reported an empty
owner for ownerless buckets, so a read-modify-write ACL update failed
with AccessDenied. GetBucketAcl and the XML check now both use the
recorded owner when the bucket has one and the resolved account
otherwise. The canned-ACL grantee moves to the same owner, and a request
carrying no account no longer erases a stored owner.
* s3: cover PutBucketAcl without auth for XML bodies and stored owners
The XML case posts the anonymous owner GetBucketAcl reports for an
ownerless bucket; the canned case asserts a recorded owner survives an
unauthenticated update and receives the full-control grant. Ambient AWS
credentials are cleared so the memory store never turns auth on.
* s3: keep the recorded owner through a PutBucketAcl update
The grantee and the stored owner were resolved differently: grants
named the recorded owner while config.Owner still took the caller's
account, so an admin updating another account's bucket rewrote the
owner and broke an XML retry naming the original one. An ownerless
bucket adopts the ACL's owner; an owned bucket keeps it.
* s3: fall back to the admin owner for an ownerless bucket in ACL handlers
An unauthenticated PutBucketAcl resolved the owner as anonymous and stored
it on the bucket, but the registry reports an ownerless bucket as
admin-owned: BucketOwnerEnforced uploads then stamped anonymous instead of
admin, and ownership checks after enabling auth compared against the wrong
owner. Both ACL handlers now resolve an anonymous fallback to the admin
account, so GetBucketAcl reports the same effective owner the registry
does and the stored owner matches it.
* s3: resolve the admin owner for bucket grants when the caller is anonymous or untrusted
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* s3: skip recording anonymous identity or account as bucket owner
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* s3: report stored bucket ACL grants verbatim
Explicitly saved anonymous full-control grants must survive a
GET-to-PUT round trip unchanged, so the read path no longer rewrites
them to the reported owner.
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* s3: expect the stored anonymous grant verbatim in GetBucketAcl
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: user.email <chrislusf@users.noreply.github.com>
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
isSuperseded only looked for chunks the event named that the filer no
longer holds. A file copied in through a mount logs a create while the
copy is still running: the mount flushes metadata with chunk uploads in
flight, so that chunk list can have holes. ChunkStreamReader stops at
the first hole, the multipart upload sends a short part ("ContentLength=
8388608 with Body length 4194304"), and since every chunk the event
names is still live the event was never superseded. The daemon retried
it forever and the offset never moved past it.
Treat chunks the filer has added since the event the same way. The
event that added them follows in the log and uploads the finished file.
* s3: read regular copy sources from their write owner
* filer: report mutation sentinel codes in ObjectTransaction responses
A failed mutation only surfaced resp.Error text, so routed writers could
not distinguish an existing-directory conflict from a generic failure and
could not apply the prefix-object retry the plain CreateEntry path has.
* s3api: retry a routed mkfile as a prefix object over a directory
mkFile retries ErrExistingIsDirectory by marking the entry a prefix
object; the routed path returned the conflict instead, so a routed write
to a key that is also a prefix failed where the lock path succeeds. The
filer now reports the code on the transaction response, making the retry
recognizable.
* s3api: commit copy destinations on the key's write owner
An owner-routed source read still left the destination on the gateway's
default filer, so a second copy resolved the first copy's destination on
the owner, found nothing, and returned NoSuchKey. Write unversioned and
suspended destinations through routedMkFile, and write the version file
on the owner the finalize recomputes on.
* s3api: cache remote objects on the filer that served the entry
An owner-routed lookup can return a remote-only entry that has not
replicated to the gateway's default filer; the cache RPC then cannot
find it and the copy or read degrades to a 503. Ask the entry's write
owner to cache it first, falling back to the default filer.
* s3: read versioned object state from the key's write owner
CopyObject destinations now commit version files and the .versions
pointer on the owner's object-scope ring key, so versioned reads through
the default filer could miss them until replication. Generalize the
routed read to dir/name lookups and paginated lists scoped to the
object's owner, and point the versioned read paths at it:
getSpecificObjectVersion, doGetLatestObjectVersion, the .versions
quick-checks in Get/Head/attributes, the self-heal rescans, tagging,
delete conditionals, directory-marker history checks, rename
destination, multipart completion checks, and the version-id format
probe. Reads fall back to the default filer when no ring owner is
resolvable, preserving single-filer deployments.
* test(s3api): fake RECOMPUTE_LATEST/DELETE and cover versioned copy reads
The ambiguous-PUT fake filer now creates parent directories on PUT
(mirroring InsertEntry), applies DELETE mutations, and re-derives the
.versions pointer for RECOMPUTE_LATEST the way applyRecomputeLatest
does, so owner-vs-replica tests can exercise versioned writes.
TestVersionedCopyDestinationReadsOnOwner copies into a versioned bucket
whose writes commit on the owner while the gateway's default filer is a
replica with no replication, then asserts reads by versionId,
latest-version resolution, and a copy chained off the new version all
resolve on the owner.
* test(s3api): gofmt the versioned-copy coverage
* s3: only swap in a cache fill of the resolved remote generation
A cache response can come back from a filer holding an older generation
of the key. Replacing the authorized source or swapping its chunks under
a size and range computed for the resolved entry serves the wrong object
or truncates the response. Match RemoteEntry identity and size before
using a cached entry, in both the copy and streaming paths.
* filer: keep directory attributes when an object lands as prefix object
The object-onto-directory retry replaced the existing directory's mode,
owner, and extended attributes with the object's. Preserve the directory
identity; merge directory-held extended keys only when the old entry
never held an object, so a prior generation's object metadata is not
carried forward.
* s3: keep marker lock checks on the filer that replaces the marker
The object-lock precondition read the key's write owner while mkdir
still replaces the marker on the default filer, so a locked marker not
yet replicated to the owner could be overwritten. Read the entry where
the replacement lands, in both PutObjectHandler and zip extraction.
Also hash a trailing-slash marker key to the same route key as the
object it names, so marker and object share one owner and one write
lock.
* s3: route tag writes to the owner tag reads resolve
GetObjectTagging reads the key's write owner, but PutObjectTagging and
DeleteObjectTagging still wrote through the default filer, so a
successful change returned the owner's stale tags until replication.
setTags, rmTags, and the versioned tag updates now write on the owner
first, falling back to the default filer when no owner resolves.
* s3: keep rename lookups on the filer that performs the move
AtomicRenameEntry and the retry token it carries both execute on the
default filer, but the source and destination reads went to the write
owner — a completed rename's token was invisible to its own retry, and
a source that only exists on the default filer looked absent. Read
both ends on the default filer.
* s3: send the .versions pointer repair with the object's route key
The heal reads the .versions entry on the object's owner, but the
conditional UpdateEntry is routed by the .versions path's own ring key,
which can hash to a different filer — leaving the pointer the routed
reads consult unchanged. Ship the CAS repair as a PATCH_EXTENDED
ObjectTransaction on the object's route key instead.
* filer: preserve only directory identity on a prefix-object write
Merging the old directory's extended attributes could carry a marker's
encryption, version, or etag fields onto the new plaintext object, since
a directory's own keys cannot be told apart from the object it held.
Keep only mode, uid, gid, user name and group names.
* s3: also require matching size when a cache fill replaces the source
The remote-generation check covers identity; size equality is the cheap
guard that keeps the streamed or copied range consistent with what was
resolved.
* s3: lock the object path for a routed .versions pointer repair
conditionalUpdateEntryRouted held the lock on the .versions path while
version writes and deletes serialize on the object path, so a repair
could pass IF_ENTRY_EQUAL and then apply over a pointer a writer had
advanced in between. Lock the object path like the writers and keep the
condition on the .versions entry via condition_key.
* s3: do not replay an owner-side tag mutation on another filer
withWriteOwnerFilerClient re-ran fn on the default filer after any owner
error, so an authoritative not-found could update a stale replica entry
and still report success, and a post-apply transport error could apply
the mutation twice. The failover walk now ends on any error fn returns;
only a channel that never came up moves to another filer.
* filer: keep a directory's xattrs when a prefix object replaces it
The prefix-object branch preserves the directory's identity but dropped
its Extended map wholesale; FUSE xattrs live there under the xattr-
prefix and belong to the directory, not to the object the marker held.
* s3: delete null and specific versions on the filer that serves them
On the default filer an absent entry deletes as success, so a version
that only reached the write owner survived a reported success while
owner-routed reads kept serving it. The suspended-versioning and
lifecycle deletes now go through routedDeleteNullVersion and
routedDeleteSpecificVersion when an owner resolves; an unsettled
attempt keeps the lock path.
* s3: route a lifecycle null-version delete to the object entry
The "null" version is the regular object entry, not a .versions file,
so it goes through routedDeleteNullVersion; only a real version id takes
the .versions-file delete.
* s3: repoint .versions only while it still names the deleted version
routedDeleteSpecificVersion recomputed the latest pointer on every
version delete, so removing a noncurrent version re-derived the pointer
to the newest remaining file — resurrecting an older version over a
cleared pointer (a suspended bucket's null write keeps it clear while
the null object stays current) or over a pointer that moved on. The
pointer is now read on the owner and repointed under an IF_ENTRY_EQUAL
guard only while it still names the deleted version; a contested pointer
is left for the self-heal path rather than overwritten by a stale scan.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* s3: let lifecycle expiry keep its metadata-only deletes on routed paths
The routed lifecycle deletes hardcoded IsDeleteData, so expiring an entry
with a volume TTL queued per-chunk deletions instead of leaving the data
for the volume to reclaim — extra work on the owner for deletes
recordMetadataOnlyIf still counts as metadata-only. Pass the caller's
cleanup choice through routedDeleteNullVersion and
routedDeleteSpecificVersion; the interactive DELETE path keeps deleting
data.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* filer: let RECOMPUTE_LATEST skip a pointer that moved on
A version delete repoints .versions only while the pointer still names the
deleted version, but the gateway cannot carry the pointer's current state
into the transaction's condition — the condition slot is already spent on
the version's WORM guards. only_if_current_name moves the check under the
transaction lock: the filer compares the pointer's name_to_key value before
scanning and skips the recompute when it moved (or was cleared by a
suspended bucket's null write), so a noncurrent delete cannot resurrect an
older version.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* s3: repoint inside the version's locked delete
repointLatestBeforeVersionDelete ran in its own transaction ahead of the
delete, so a version delete denied by retention or legal hold still moved
the latest pointer — GET and HEAD went on serving the older version it
rolled to. The repoint and the delete now ride one transaction behind the
version's WORM guards, atomically gated by the new only_if_current_name
recompute guard instead of a stale-read IF_ENTRY_EQUAL.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* vector: Lease::forget hands the tag name off without releasing
A distributed planner pins the planned version and forwards the tag
name to the executor — the guard must not release on drop.
* vector: forget untracks the lease so an abandoned handoff expires
The tag left LIVE_TAGS but the drop-time release was disabled — an
abandoned handoff survived forever locally. Untrack it so the lease
TTL still reaps it.
* vector: DeleteIndex format guard, sticky discovery, advertise normalize
- handleDeleteIndex verifies the table carries the Lance format before
DeleteTable — an Iceberg table under the same namespace is not an
index and must not be removed by it.
- discovery keeps the current worker while it is still registered —
membership-order churn can no longer cancel in-flight calls, and the
resolver closes each master conn inside a 5s deadline.
- advertised_addr strips scheme and .grpcPort so the UDP probe reaches
every supported master form.
* vector: empty-follower retry, format check without read grant
- discoverVectorWorkers keeps looking past an empty master reply —
followers list nothing when workers registered with the leader.
- the DeleteIndex format check uses the internal catalog read, so
DeleteTable-only callers don't gain a GetTable requirement, and the
format must be Lance exactly — format-less tables no longer pass.
* ec: let shard-interval recovery exclude a set of suspect shards
gatherEcShardIntervals and recoverOneRemoteEcShardInterval skipped only
the shard being rebuilt. A needle-level rebuild wants every suspect
shard kept out of the gather so a bad copy cannot poison the
reconstruction it is being compared against.
* ec: rebuild a needle's suspect shard ranges when it fails to parse
A bit-rotted data shard serves a clean-looking but wrong interval, so the
missing/unreadable-shard fallback never engages and the needle CRC check
in ReadEcShardNeedle returns the corruption to the caller. Each interval's
source shard is a suspect: rebuild the suspect ranges from the surviving
shards — all of them at once while parity covers the suspect set, or one
at a time when the needle spans more shards than that — and let a
successful needle parse arbitrate which copy was corrupt. Shards whose
range changed are dropped from the location cache so later reads go
straight to reconstruction.
* ec: mirror the corrupt-shard needle rebuild in the Rust volume server
The snapshot now keeps every interval's shard placement so a parse
failure can name its suspects. rebuild_ec_shard_needle rebuilds the
suspect ranges through recover_one_remote_ec_shard_interval with the
suspect set excluded from the gather, and the winning candidate's
differing shards are dropped via forget_shard_location.
* ec: recover suspect ranges under the requested needle id
A failed ReadBytes re-reads the id out of the possibly corrupt header, so the rebuild passed whatever it decoded to peer recovery requests; carry the requested id through instead, and reject a candidate whose parsed id does not match it.
* ec: retry a failed needle rebuild with each gathered input dropped
The gather behind a suspect-range rebuild can itself serve a second corrupt shard: Reed-Solomon then reconstructs wrong bytes and the needle parse rejects the only candidate. Gather every surviving shard once, then re-decode with each of the first decode's inputs dropped in turn; the parse arbitrates the honest subset.
* ec: mirror the dropped-input rebuild and id check in the Rust volume server
Same two guards as the Go side: a candidate that parses under a different needle id is rejected, and a failed candidate whose gather fed it a corrupt input is retried by dropping one gathered shard at a time. The gather is shared through a want-bounded gather_ec_shard_intervals so the retry pays no extra fan-out.
* ec: answer deleted when a rebuild's holder reports the needle deleted
The rebuild discarded the deleted flag its interval recovery collected, so a needle deleted on its other holders but live in the local index came back as a corruption error. A deleted report is authoritative, so it now short-circuits the hypothesis search and answers ErrorDeleted.
* ec: hydrate rebuild candidates into a scratch needle
A failed ReadBytes leaves partially parsed fields on the needle, and optional sections do not reset when absent, so a corrupt first parse could leak stale name/mime/ttl into a successful candidate. Parse each candidate into a fresh needle and adopt it only on a matching id.
* ec: corrupt parity+1 shards in the exceeds-parity read test
The loop skipped the suspect's own shard when it fell inside the parity range, leaving only ParityShards corrupt shards — a case recovery can legitimately fix. Count the additional corruptions so the test always exercises the boundary it names.
* ec: mirror the deleted-report short-circuit in the Rust volume server
A holder's deleted answer during a rebuild propagates out as EcMiss::Deleted instead of surfacing as InvalidData, matching the Go path's authoritative-delete rule.
* master: replicate the recovery-wait growth unfreeze through raft
state_loss_recovery_wait lifted the volume-growth freeze only in
memory, while the freeze itself was replicated. A restart replayed the
frozen flag and growth stayed refused until the wait expired again, on
every restart, unless an operator ran volume.growth.resume.
The lift now applies GrowthFrozen=false like ResumeVolumeGrowth, from a
goroutine since IsGrowthFrozen is a read path that cannot block on
raft. The apply runs under reportedTopologyIdsLock so a freeze landing
after the lift keeps its command last in the log; the stale unfreeze
skips itself when the flag is frozen again.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* master: cover raft replication of the recovery-wait unfreeze
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* master: retry and re-arm a growth unfreeze raft failed to record
applyGrowthFreeze swallowed the raft error, so a rejected lift during a
leader change left the flag clear while the log still held the freeze;
nothing retried and a restart re-froze growth anyway. The apply now
returns the error, replicateGrowthUnfreeze retries it, and on
persistent failure re-arms the flag so the next IsGrowthFrozen
attempts the lift again instead of diverging from the log.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* master: release the topology-id lock before the freeze raft write
CheckReportedTopologyId and SetTopologyInitPending held
reportedTopologyIdsLock across the blocking raft apply, so a slow
apply stalled every later report — including the heartbeat that
should trigger the next freeze. The lock now covers only the
mismatch decision; the replicate step runs after it is released.
* master: bind the recovery unfreeze to the freeze instant it lifts
An unfreeze submitted while a newer freeze sat committed but not yet
applied — e.g. after this node won an election — could land behind
that freeze in the log and erase it. The command now carries the
freeze instant cluster-wide (GrowthFrozenAt) and the lift is bound
to the instant its submitter observed (GrowthUnfreezeFor): the state
machine no-ops a lift while a strictly newer freeze exists, which
also lets the unfreeze run without holding the topology-id lock at
all. Concurrent freeze writes keep their own mutex so log order
cannot regress the instant.
* master: a superseded unfreeze retry leaves the resumed flag alone
replicateGrowthUnfreeze broke out of its retry loop when a newer\nfreeze superseded the lift, but the break fell through to the\nre-arm: an operator resume in between was silently undone. Return\ninstead — the newer freeze generation owns the flag.
* master: replayed freeze entries without an instant stay ageless
Freeze commands written before the instant field existed carry zero.\nSubstituting the replay time made each restart stamp the freeze as\nnew, so a committed bound unfreeze always looked older and was\nrejected — the freeze came back on every restart. Keep the zero\ninstant: the recovery window reads as elapsed and bound lifts apply.
* master: pack the growth freeze flag and generation into one atomic
Separate flag and timestamp atomics left three holes: a resume's\nload-and-clear could lift a generation stamped in between, a bound\nlift compared stamps numerically although masters' clocks need not\nbe monotonic, and an ageless generation had no identity a bound lift\ncould match without colliding with the unconditional encoding.\n\nOne value now carries the state — 0 unfrozen, -1 an ageless freeze,\nelse the freeze instant — so every transition is a single atomic\nswap or compare-and-swap bound to an exact generation.
* master: a repeat mismatch report keeps the armed freeze generation
Re-stamping while already frozen never reaches the raft log, so the local generation diverges from the replicated one and every bound unfreeze becomes a no-op on replay — the freeze returns after restart or failover.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* master: snapshots read the freeze flag and generation in one load
Save, Snapshot, and PersistMaxVolumeId read IsGrowthFrozen and the generation separately, so a resume landing between them persisted frozen=true with stamp 0 — restoring an ageless freeze a committed bound lift cannot clear.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* iceberg: serve a registered table's metadata on load and commit
RegisterTable saved only the metadata location, and loads and commits
never read the file there. They served a placeholder with no schema, no
snapshots and a new random table UUID on every load, so a registered
table read as empty, an INSERT failed against the empty schema, and
every other commit failed its assert-table-uuid requirement. Metadata
uploaded through the S3 gateway, as a migrated table's is, is stored in
chunks that registration did not read, so it registered an empty table.
Read the file following its chunks, reject one that is not Iceberg
metadata with a 400, and keep its content and table UUID in the record
the way CreateTable does.
Records registered before this change hold only the location. Loads and
commits now read the file there, and the first commit stores it in the
record under the file's table UUID.
* iceberg: transactions read a location-only registration's metadata file
* iceberg: cover transactions on a location-only registration
* iceberg: refresh the default name mapping on transaction commits
* iceberg: cover name-mapping refresh in transactions
* iceberg: roll a transaction back to a registration's metadata file
A transaction that failed after flipping a table registered before the
record kept its metadata could not undo the flip. The rollback restored
the record's prior metadata, which for such a table is only a location,
and UpdateTable keeps the full metadata a flip stored when the restore
sends none. Loads went on serving the change the client was told had
failed, and a client retrying the transaction could apply it twice.
Restore the metadata loads served from the file instead, under the
file's table UUID.
* iceberg: register oversized metadata by its location
Registration sends the metadata file's content to the S3 Tables
manager, which refuses a request over its 10MB body limit, so a table
whose metadata is larger could no longer be registered at all.
Register such a file by its location and table UUID alone; loads read
it from that location, as they do for a record that never kept its
metadata. Export the limit as s3tables.MaxRequestBodySize so the
catalog can check against it. A commit to such a table still sends the
whole metadata to UpdateTable and stays over the limit.
* iceberg: commit oversized metadata by its location
* iceberg: let maintenance follow a location-only table record
* admin: read a table's metadata file when the record keeps none
* iceberg: bound the metadata file a location-only record reads
* iceberg: bound only chunked metadata reads in the worker
Inline content is what the catalog itself wrote and the REST load path
accepts it at any size, so rejecting it in loadMetadata stranded tables
over 64 MiB from every maintenance scan. Cap only the chunked read, where
the recorded location can name any object in the bucket.
---------
Co-authored-by: chrislusf <chrislusf@users.noreply.github.com>
* sftp: fail rm and rmdir of a missing path, and mkdir of an existing one
* sftp: create directories exclusively
---------
Co-authored-by: user.email <chrislusf@users.noreply.github.com>
* topology: add AvailableSpaceWithinLocation for location-scoped free slots
AvailableSpaceFor answers for the whole topology; a location-constrained
assign needs to know whether the requested data center, rack, or data node
still has room before deciding growth can satisfy it.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* master: let location-constrained assigns wait for pending growth
An assign carrying a data center or rack constraint failed on the first
unsatisfied pick, even when the same request had just queued volume growth
for that location — so the first write into a fresh collection errored out
instead of picking up the volume growth landed a moment later.
Keep the fail-fast path only when growth cannot help: growth disabled, no
growth pending, or no free slots left inside the constraint. Otherwise the
request takes the same wait/shed path as an unconstrained assign.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* topology: sum leaf free slots in AvailableSpaceWithinLocation
Rolled-up rack and data center counters keep a maintenance-mode server's
capacity, and one overcommitted node can cancel a sibling's slots. Sum the
matching data nodes directly so the location check agrees with what growth
can actually use.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* master: let constrained growth initiators reach the shed path
When a constrained assign's own growth concluded unfulfilled, the fast-fail
check saw only that no grow was pending and returned the plain pick error —
a code the assign client does not retry, so the first write died instead of
re-triggering growth. Exclude the request's own initiated grow from the
no-pending-growth condition so it reaches the existing re-pick and
ResourceExhausted shed like an unconstrained assign.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* master: judge targeted grow requests by the location they name
A grow request carrying a data center, rack, or data node constraint was
discarded whenever the layout's writables elsewhere were healthy, so a
location with free slots but no writable could never receive the volume its
assign was waiting for. Add ShouldGrowVolumesWithinLocation and use it for
constrained requests; the layout-wide check still governs untargeted ones.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* topology: only positive leaf slots count in AvailableSpaceWithinLocation
An over-committed node reports negative free slots; summed plainly it\ncancelled a sibling's room and a constrained assign could fail fast\nwhile the location still had space to grow into.
* master: fail constrained assigns fast when growth cannot place
A constrained assign with free slots in its location still waited out\ngrowth that could never satisfy the replica placement — a single-rack\ndata center asked for 010 sheds the same doomed grow on every client\nretry. Run the grow-time node search as a dry run and return its\nplacement error instead of a retryable shed.
* master: judge constrained growth by layout, not free slots
A slot-level feasibility probe races with the grow it judges: a\npending grow consuming the last slot in one rack reports the\nplacement impossible while it mints the request's own volume, and\nthe hard error skips the re-pick that would find it. GrowthBlocker\nnow counts racks and data nodes inside the named location — shape a\ngrow cannot consume — so only genuinely unplaceable layouts fail\nfast.
* topology: GrowthBlocker mirrors the grow selector's feasibility
The location filter bound every replica tier, so a 010 request naming one rack needed two racks inside it while the selector constrains only the first placement and counts the rest by room. And counting nodes without free slots let a full location queue a grow it could never serve, so the assign shed ResourceExhausted at timeout instead of failing fast. GrowthBlocker now applies the option's location to the primary placement chain only and leaf-sums free slots the way the selector does, so it reports a blocker exactly when the grow can never place.
* master: repick once before surfacing a growth blocker
An in-flight grow can spend the last free slot minting the request's own volume: the blocker's slot counts predate that registration by a heartbeat, so returning the placement error on the first failed pick could reject a write whose volume lands a moment later. A pending grow now earns one repick grace before the blocker error is returned.
---------
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* master: end the post-mint allocation hold on the embedded reporter's clean inventory
DeferAllocation armed a warmup-length hold whenever the leader minted a
fresh topology id, and on weed server the hold ran the whole window: the
embedded volume server's join kept refreshing the quiet evidence clock, so
the two-pulse early release still cost the first write 10-15s on every
fresh start. weed mini already releases on its single pinned reporter's
first inventory, but pinning refuses outside volume servers, which weed
server must keep admitting.
Add a non-exclusive variant: the topology names one expected reporter
whose first clean full inventory lifts the hold, while other reporters
stay admitted and still feed prior-state evidence and the topology-id
freeze. A single-master weed server points it at its embedded volume
server — the one reporter guaranteed to arrive quickly; a veteran's
heartbeat carries prior state and keeps the window through the regular
checks, so a wiped -dir with surviving external volume servers still
trips the freeze on first report. Restarted clusters are untouched: a
recovered topology id means no mint and no hold.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* filer: announce on the startup peer scan for embedded single-master setups
A fresh filer whose startup peer scan came back empty re-polled for the
full confirm window before registering on the master, so the lock ring
had no member and routed traffic such as S3 PUTs spun on "no lock server
found" for ~10s on every fresh weed server / weed mini start.
For setups that start with exactly one filer the scan is already final:
a single-master weed server and weed mini announce as soon as it
completes. A late external filer still joins and replays through the
normal path — the confirm window only ever caught peers arriving inside
it, and the residual risk mirrors the accepted volume-side trade-off: an
embedded filer announcing a few seconds earlier can own the lock ring
while a surviving external filer's backlog is still on its way.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* volume: retry a failed heartbeat promptly until first registration
A heartbeat stream refused before the leader is known — the common race
when a volume server starts while an election is still running — always
fell back to sleeping out the full pulse, so a fresh volume's first
registration could wait ~5s behind a leader elected a second later.
The peer refresh already identifies the leader during the wait, so cap
the retry pause at a second once a leader is named; while no leader is
known the same short retry applies until the first stream is confirmed,
since an unregistered server has nothing else to do. Once the server was
registered the pulse backoff is unchanged, so a lost leader still paces
reconnects the way it did before.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* volume: retry a failed heartbeat promptly until first registration (rust)
Mirror the Go heartbeat retry: cap the post-refresh wait at a second when
a leader is named, and keep the short retry until the first stream is
confirmed so a fresh volume registers promptly when it races an election.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* master: leave admitted reporters a confirm pulse when the embedded report ends the hold
The single-master weed server releases its post-mint allocation hold on
the embedded volume server's clean report, but outside volume servers
stay admitted and one may still carry prior cluster state — an inventory
from the embedded server alone cannot prove none is due. Clearing the
hold outright let a rebuilt master re-issue a volume id a reconnecting
veteran still owned.
Collapse the released hold to one heartbeat interval instead: late
reporters land their evidence or a topology-id freeze inside the pulse,
and a re-mint re-arms that pulse rather than the full warmup. The pinned
single-reporter release (weed mini) still drops the hold outright, since
no other reporter can be admitted.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* filer: keep the empty-membership confirm on embedded single-master weed server
A single-master weed server still admits external filers, so an empty
startup peer scan is not final: after a master restart the existing
filers simply have not reconnected yet. Announcing on the scan let the
embedded filer take lock-ring ownership and serve writes that a peer's
replayed backlog could then overwrite.
Only weed mini keeps the skip, where the embedded filer is the only
member its dev cluster can have. The confirm window only delays the
master listing for a fresh store; the filer serves while it runs.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* volume: tolerate an expired heartbeat retry deadline (rust)
refresh_leader_from_peers can consume the whole pulse budget, leaving
retry_at in the past; subtracting a later Instant panics and kills the
heartbeat task instead of retrying. Match the Go side, where
time.Until of a past deadline just sleeps zero.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* volume: retry the seed list without a second pulse sleep (rust)
The error path already slept its retry delay; an extra full pulse after
losing the round delayed the next attempt past what Go's heartbeat()
does, which paces retries only inside the per-master attempt.
* master: keep a configured allocation hold shorter than the confirm pulse
Once the embedded report lands, the collapsed hold replaced the
configured deferral with a full pulse even when the operator set a
shorter one, including zero (disabled). Cap at the pulse instead.
* filer: end the empty-membership confirm early on a fresh-minted embedded master
On a fresh single-master weed server the announce gate still held the
filer for the full confirm window, so the master's lock ring stayed
empty and the first write retried 'no lock server found' for ~7s on
top of the collapsed allocation pulse. A fresh TopologyId mint proves
the master's persisted state was empty, making the empty peer scan
final; skip the rest of the confirm only then, and keep it whenever
the master recovers a prior id.
* filer: keep a short reconnect window on a fresh-minted embedded master
A fresh TopologyId mint proves the master's persisted state was empty,
not that no external filer survived its loss — one can still be
mid-reconnect on its keepalive cycle when the confirm starts. Ending
the empty-scan confirm on the first poll let the fresh filer register
and serve routed reads before that peer was found and backfilled, so
existing files could read as missing. The fresh-mint path now shortens
the confirm to a few keepalive cycles instead of skipping it: a
reconnecting filer still lands in time, while a genuinely fresh cluster
announces after ~3s rather than ~10s.
* volume: skip the heartbeat loop when no seed masters are configured
An empty seed list made the outer loop spin with no await between
passes and no path to observe the stop flag. A masterless volume server
has nothing to register with, so the heartbeat exits after a warning.
* volume: skip the heartbeat loop when no master addresses are given (rust)
Mirrors the Go heartbeat: an empty seed list spun the outer loop with
no await and no shutdown checks. Exit after a warning so library
callers get the same behavior as the binary path that validates the
list up front.
* filer: re-check the fresh-mint hint on each confirm pass
The topology id is minted after raft election, which can land after
the empty-scan confirm has already started — a one-time check at entry
missed the mint and held the full window. Evaluate the hint on every
poll and shorten the confirm from the first observed mint, still
capped by the normal confirm deadline.
* master: mark a fresh topology mint only when the volume bound is zero
Recovery restores the max volume id unconditionally but the topology id
only when present, so an upgraded or partially rebuilt master can mint a
new id while still holding prior volume state. Marking fresh there would
let the embedded filer shortcut its peer confirm on a cluster that is
not empty; only mark the mint fresh when no bound survived.
* filer: drop the fresh-mint shortcut once a peer has been observed
---------
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* s3: allow PutObjectAcl when auth is off
* s3: ownerless objects claim the admin account when auth is off
An object written through the filer carries no owner metadata, so
objectOwner fell back to an empty account id and ExtractAcl rejected the
canned ACL's canonical grantee. Extend TestPutObjectAclWithoutAuth to
cover the ownerless case.
* s3: GetObjectAcl reports the admin owner for ownerless objects when auth is off
The same empty-owner fallback produced an ACL response with a blank
Owner ID and a blank canonical grantee.
* s3: ignore the client-supplied account id when auth is off
Auth does not strip AmzAccountId on the iam-disabled path, so the
ownerless fallback could persist a forged owner. requestAccountId
substitutes the admin account, matching the idiom already used in the
upload ACL path.
---------
Co-authored-by: user.email <chrislusf@users.noreply.github.com>
* vector: only follow a leader hint that points elsewhere
The master sends volume_location on every KeepConnected response —
the leader always names itself, so the worker exited the stream on
the first response and flapped every 3s, leaving ListClusterNodes to
see the node intermittently. Only redirect when the hint points at a
different master.
* vector: leader hints carry host:http.grpc — normalize before comparing
A self-hint arrives as host:9333.19333 while --masters gives host:9333,
so the raw comparison still redirected and grpc_addr could not parse
the dotted port. grpc_addr now prefers the explicit .grpcPort, and the
hint comparison normalizes both sides to host:httpPort first.
* vector: owner-prefixed lease tags; foreign leases get a long backstop
reap_expired exempted only process-local LIVE_TAGS — server B could
reap server A's live query lease at the 1h TTL. Tags now carry a
per-process instance id: own leases use the short TTL, foreign leases
only the 24h backstop, so a live query elsewhere keeps its pin while
dead-process tags still collect eventually.
* vector: pick the lease-tag format by segment count, not numeric-ness
An owner that is all decimal digits parsed as the legacy timestamp, so
its real timestamp was ignored and the lease never expired. The third
segment is what distinguishes {instance}-{ts}-{uuid} from {ts}-{uuid}.
vector: cover the index name in named_segment_fragments
The planner groups tasks by index name — assert the helper actually
returns the logical name so a regression dropping it fails.
vector: cover the identity object in the s3vectors auth context
The auth middleware must carry the resolved identity object — only
the name survives otherwise and authorization loses actions, account,
and claims.
* sftp: refuse mkdir and uploads under a missing folder
* sftp: re-check the parent directory when a buffered upload commits
The open-time check cannot catch a parent removed while the upload
buffered; the filer would recreate it when the entry lands.
* sftp: cover mkdir and upload commit against a missing parent
Extend the test filer with CreateEntry so makeDir can be exercised end
to end.
* sftp: refuse a rename into a missing folder
The filer's rename path creates the target entry through CreateEntry,
which recreates absent parents the same way uploads did.
---------
Co-authored-by: user.email <chrislusf@users.noreply.github.com>
* s3vectors: cover non-filterable key round-trip and nested filter rejection
The catalog writes the key list as a JSON string while older records
may still carry the array form; indexFromRecord must decode both. The
rejection path extracts field names through nested $and/$or filters, so
a blocked key deep in the tree still fails the query.
* s3vectors: factor the non-filterable rejection into a testable gate
Pull the blocked-key check out of handleQueryVectors into
nonFilterableField so the rejection itself is covered without standing
up the catalog stack; the handler now 400s on whatever the gate names.
* s3vectors: report the non-filterable match separately from the key
An empty metadata key in nonFilterableMetadataKeys collided with the
"no match" sentinel, so a filter on that key skipped rejection. Return
a match flag and cover the empty-key case in the test.
* wdclient: cover a lapsed caller deadline that Err() has not observed yet
* wdclient: reuse isCanceledErr for the elapsed-deadline check
* wdclient: reinterpret only deadline errors as caller-gave-up
isCanceledErr also matches gRPC Canceled, which a filer can return for
its own reasons; treating it as an elapsed caller deadline hid the
failure from the filer's health record. The fallback now matches only
context and gRPC DeadlineExceeded, so filer-side cancellations keep
counting as filer failures.
vector: non-filterable metadata round-trips and filters reject it
- GetIndex omitted nonFilterableMetadataKeys — the catalog stores the
list as a JSON string but the reader only accepted an array; both
forms now decode.
- QueryVectors now rejects filters referencing non-filterable keys
instead of silently scanning them.
* vector: weed-vector registers with the weed master
weed-vector-server --masters holds a KeepConnected stream to the weed
master, the same mechanism filer and s3 use via wdclient — the server
shows up in ListClusterNodes(client_type=vector). Resends registration
and rediscovers on failure.
* vector: s3 discovers weed-vector through the master
With -vectors.grpc unset the vectors gateway resolves one registered
weed-vector via ListClusterNodes on the masters it already derives
from filer config — discovery replaces the static dial for the
single-worker OSS path. The conformance harness registers the engine
to mini's master and drops -vectors.grpc.
* vector: default-fill registration fields
..Default::default() keeps the source compiling against proto variants
that carry extra fields, so the same file works in either tree.
* vector: serve the Vectors API on the main s3 port
The vectors router sits on the main -port listener behind a dispatch
that checks the SigV4 credential scope for /s3vectors/ — no separate
port needed. -port.vectors stays for explicit dedicated-port
deployments; the conformance harness now hits vectors through :8334.
* vector: drop -port.vectors entirely
It was new — no compat to preserve. The main -port listener serves the
API whenever a weed-vector backend is reachable.
* vector: drop -vectors.grpc; discovery is the only locator
weed-vector always registers to master, the gateway always resolves it
through ListClusterNodes — a static address just forks membership.
* vector: binary is weed-vector
Matches the enterprise binary; package and lib names are unchanged.
* vector: review fixes — cluster type, grpc port, lazy resolve, leader hint, advertise
- cluster: register 'vector' as a node type so ListClusterNodes returns
weed-vector members
- s3: dial masters via ToGrpcAddress (HTTP port + 10000)
- s3: backend resolves lazily through vectorDiscovery — workers joining
after gateway startup work on first query, and address changes re-dial
- master_client: convert master HTTP addr to grpc +10000, honor http(s)
scheme, follow the leader hint in KeepConnectedResponse, and
--advertise covers wildcard/unreachable listen addresses
* vector: no --advertise flag; derive the outbound address
A wildcard listen resolves to the source IP the master sees via a
route lookup, so registration always advertises something reachable
without another flag.
* vector: review fixes — plaintext dispatch, conn leaks, discovery backoff
The two plaintext http servers now serve the vectors dispatch too —
signed s3vectors calls on the main port reach the right router.
discoverVectorWorker closes each master conn and bounds lookups to 5s;
a dead or absent worker no longer re-hits master on every RPC — the
resolver backs off for 5s between rediscovery attempts.
* vector: 1 GiB grpc cap, dead-worker invalidation, CLI rename
Valid PutVectors batches exceed the tonic 4 MiB default — match the
gateway's 1 GiB cap. A replaced worker no longer sticks: any RPC error
resets the cached address and the next call rediscovers through the
master. The core CLI bin renames to weed-vector-cli so the two crates
do not collide on the same output path.
* vector: retry-only-transport reset, periodic re-resolve, cli rename
- reset() takes the failed client and runs only on transport errors —
an InvalidArgument on a healthy conn no longer interrupts unrelated
requests, and a concurrent re-resolve wins the race.
- cli() re-resolves once the interval passes even with a live client,
so a worker that moves addresses is picked up.
- verify.rs spike docs use weed-vector-cli after the bin rename.
* vector: only Unavailable counts as a transport failure
A request deadline or an application error can fire on a healthy
connection — resetting the shared conn on those churned reconnects.
vector: keep typed error chains in lease pin/release
Error::msg discarded the source chain — denied() could not downcast to
object_store::PermissionDenied, so try_pin surfaced the error instead
of degrading to an unleased query and read-only credentials broke
GetVectors/QueryVectors. Plain ? preserves the chain.
* wdclient: treat an elapsed caller deadline as caller-gave-up on lookup errors
gRPC can surface its own DeadlineExceeded a tick before ctx.Err() becomes
visible, so a lookup that died on the caller's deadline was sporadically
recorded as a filer failure (seen in the linux/386 job).
* wdclient: reinterpret only deadline errors against an elapsed caller deadline
vector: named_segment_fragments carries the logical index name
Sharded builds create one named index per shard and a
with_index_segments scan must not mix names — a fanout planner needs
the name to group tasks, so expose it alongside uuid and coverage.
vector: keep the full identity in the request context
Auth discarded the resolved identity object — the s3tables manager
request then re-resolved by name and denied authenticated users their
actions, policies, account, and claims. Store the object via
SetIdentityInContext so downstream authorization sees it.
* build(deps): bump com.google.guava:guava in /test/java/spark
Bumps [com.google.guava:guava](https://github.com/google/guava) from 32.1.3-jre to 33.7.1-jre.
- [Release notes](https://github.com/google/guava/releases)
- [Commits](https://github.com/google/guava/commits)
---
updated-dependencies:
- dependency-name: com.google.guava:guava
dependency-version: 33.7.1-jre
dependency-type: direct:production
...
Signed-off-by: dependabot[bot] <support@github.com>
* test/java/spark: bump guava to 33.7.2-jre
33.7.1 is covered by a published advisory for uncontrolled memory
allocation during CompactHashMap/CompactHashSet deserialization;
33.7.2 is the patched release.
* test/random_access: bump guava to 33.7.2-jre
Same deserialization advisory; the pin predates the fix.
---------
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: user.email <chrislusf@users.noreply.github.com>
* helm: render the filer's mysql credentials once when the user sets them
With the MySQL filer store, the filer reads WEED_MYSQL_USERNAME and
WEED_MYSQL_PASSWORD from the chart's db secret. Setting either one in
extraEnvironmentVars or secretExtraEnvironmentVars, as helm_ci.yml does for
the password, added a second entry with the same name: Helm 4 rejects the
install (env: duplicate entries for key), and Helm 3 deploys with a warning
that one definition may be dropped on apply.
The db secret's entry is now left out for a key the user sets, so each key
is rendered once and the user's value wins, the rule #8488 applied to a key
set in both global and filer extraEnvironmentVars. Without an override the
filer StatefulSet is unchanged.
* helm: render the selected filer mysql credential ahead of the merged vars
A credential the user sets now renders in the early position the db
secret's entries used to occupy, and is excluded from the merged and
secret loops. Kubernetes resolves a $(VAR) reference only against names
declared earlier, so a later entry such as WEED_MYSQL_DSN can again
reference the active credential. Secret-backed values win over plain
ones; the db secret's entry remains the fallback.
* helm: let a secret-backed filer env key replace the plain one
secretExtraEnvironmentVars now wins over extraEnvironmentVars for the
same name, rendering once at the merged key's sorted position instead
of twice. The values.yaml WEED_MYSQL_* defaults no longer duplicate
when overridden through the secret map - e.g. enabling the store via a
secret WEED_MYSQL_ENABLED also rendered the "false" default before,
which Helm 4's server-side apply rejects as a duplicate.
* helm: keep a credential that references another env var in sorted position
A user-supplied credential whose value itself holds a $(...) reference
stays in the merged vars' sorted order so its own references still
resolve; only reference-free values, valueFrom entries, and the
db-secret defaults move to the early slot.
* helm: render vars a filer mysql credential references ahead of it
Hoisting the selected WEED_MYSQL_USERNAME/PASSWORD entries above the
merged extra vars let WEED_MYSQL_DSN-style $(...) references resolve,
but it broke the opposite direction: a user-set credential such as
WEED_MYSQL_USERNAME: "$(DB_USER)" was emitted before DB_USER, so
Kubernetes left the reference literal. Pull every merged or secret var
a credential value transitively references out of the sorted set and
emit them first, dependencies before dependents, so both directions
resolve.
* helm: emit filer mysql credentials after the chart's own env vars
The dependency-ordered credential block rendered before
SEAWEEDFS_FULLNAME and the s3 credentials, so a user-supplied
WEED_MYSQL_USERNAME could not reference them via $(...). Move the
block below those fixed entries so references in either direction
resolve.
* helm: bound filer mysql credential ordering passes by var count
Chains deeper than the fixed eight passes forced a credential out ahead
of the vars it references. Cap discovery at the number of merged/secret
vars and emit passes at the ordered list length, so any resolvable chain
finishes before the last-pass fallback emits cyclic leftovers.
---------
Co-authored-by: chrislusf <chrislusf@users.noreply.github.com>
* helm: keep the filer db credentials across uninstall and install
The db secret is kept on uninstall (`helm.sh/resource-policy: keep`), but it
is a pre-install hook, and Helm deletes an existing hook resource before
creating it again. The next install therefore wrote a new random password
and reset the user to YourSWUser, so the filer could no longer log in to a
database set up with the old ones, including credentials edited into the
kept Secret.
The template now reads the existing Secret with lookup and reuses its user
and password; a first install still generates them as before. A kind step
in helm_ci.yml installs, edits the Secret, uninstalls, installs again and
checks both values.
* helm: README, the db secret is kept and reused across a reinstall
* helm: make the db secret hook-delete-policy explicit
before-hook-creation is Helm's default for hook resources, but spell it
out so the reuse contract is obvious: lookup reads the kept Secret at
render time, the old copy is deleted before the hook recreates it, and
the stored credentials carry over.
---------
Co-authored-by: Chris Lu <chris.lu@gmail.com>
Only a string rendered as `value`; anything else went to `valueFrom`. An
unquoted number or boolean in a values file or `--set`, such as
`WEED_MYSQL_PORT: 3306` or `WEED_MYSQL_ENABLED=true`, rendered as
`valueFrom: 3306` and the API server rejected the workload.
A map still renders as `valueFrom`, a null still renders an empty variable,
and strings are unchanged. Numbers go through toJson because values files
decode them as float64, and quote would print 1000000 as 1e+06.
* filer: make the HTTP connection idle timeout configurable
The filer applied a hardcoded 10s activity timeout to its HTTP
listener. A buffering proxy in front of the filer can stall upstream
reads longer than that when the end client drains slowly, so large
downloads through it were truncated (#8062).
Expose the value as -idleTimeout (default 10, unchanged), matching the
existing -idleTimeout flags on the volume and s3 servers, and wire
filer.idleTimeout through 'weed server' and 'weed mini' too. A nil
option keeps the 10s default so embedders constructing FilerOptions
directly keep working.
* filer: apply the idle timeout to the readonly port too
The -port.readonly listener ran without any connection timeout while
the main API port timed out. Give it the same -idleTimeout so both
ports behave identically.
* webdav: make the HTTP connection idle timeout configurable
The WebDAV server used the same hardcoded 10s activity timeout.
Expose it as -idleTimeout (default 10, unchanged), including
-webdav.idleTimeout when embedded via 'weed filer -webdav',
'weed server -webdav', and 'weed mini'.
* iam: make the HTTP connection idle timeout configurable
The IAM API server used the same hardcoded 10s activity timeout.
Expose it as -idleTimeout (default 10, unchanged), including
-iam.idleTimeout when embedded via 'weed filer -iam' and
'weed server -iam'.
* sftp: make the connection idle timeout configurable
The SFTP server used the same hardcoded 10s activity timeout.
Expose it as -idleTimeout (default 10, unchanged), including
-sftp.idleTimeout when embedded via 'weed filer -sftp' and
'weed server -sftp'.
* command: clamp idle timeout seconds before converting to duration
A seconds value past time.Duration's range would overflow on
multiplication and silently disable the timeout. Bound it in one shared
helper, which also deduplicates the nil-default handling at the four
call sites.
* command: return zero for non-positive idle timeout seconds
A large negative seconds value could wrap on multiplication and come
out positive. Guard before converting so non-positive input always
disables the timeout.
* command: apply idle timeout to local unix socket listeners
The unix-socket listeners in filer, s3, and sftp served accepted
connections with no activity deadline while the TCP listeners already
wrapped conns in util.Listener. Wrap the socket listeners the same way
so a positive idleTimeout also bounds idle socket connections.
* command: clamp s3 idle timeout seconds before multiplying
s3 still multiplied the flag inline, so an out-of-range value could
overflow time.Duration. Route it through idleTimeoutSeconds like the
other commands; non-positive keeps meaning disabled.
* volume: clamp idle connection timeout seconds before converting
* command: cover idleTimeoutSeconds conversion semantics
* command: gate idleTimeoutSeconds clamp check on 64-bit int
* s3: answer NoSuchBucket for encryption and policy writes to a missing bucket
PutBucketEncryption, DeleteBucketEncryption, and PutBucketPolicy surfaced
the filer's bucket-miss as InternalError; GetBucketEncryption masked it as
ServerSideEncryptionConfigurationNotFoundError. Map the miss to
NoSuchBucket instead, matching AWS. Internal readers still treat a missing
bucket as "no encryption configured" so uploads keep their behavior.
* s3: wrap ErrBucketNotFound only for verified misses in bucket policy ops
The filer lookup helpers in get/set/deleteBucketPolicy marked every
LookupDirectoryEntry failure as bucket-not-found, so a transient filer
outage answered NoSuchBucket for a live bucket. Route them through
filer_pb.LookupEntry, which normalizes only true misses to ErrNotFound.
* s3: answer NoSuchBucket when the policy delete's own write loses the bucket
* s3: classify a filer NotFound on the policy UpdateEntry as ErrBucketNotFound
* s3: require the bucket entry to exist on config patch transactions
* s3: answer NoSuchBucket when tagging and CORS writes lose the bucket mid-request
* s3: keep the existence check on no-op bucket config patches
An unchanged config write produced an empty patch that early-returned
before the transaction, so a bucket deleted between the read and the
write still reported success. Ship the condition-only transaction so
IF_EXISTS runs on its own.
* s3: guard the iam step and cover failure-vs-miss classification
Nil-guard s3a.iam in the bucket policy handlers so the handlers do not
panic when no IAM server is wired (unit tests, embedded servers), treat
a "precondition failed" response string the same as the error code when
mapping a failed patch to ErrNotFound, and add handler-level tests that
pin the miss/failure distinction: missing buckets and vanished mid-write
buckets answer 404 NoSuchBucket while transient filer failures stay 500
InternalError.
* s3: keep multipart part encryption on the initiate-time decision
putToFiler applied the *current* bucket default encryption when writing a
staged part, so a PutBucketEncryption landing between two UploadPart calls
produced an object whose parts disagree: some chunks ciphertext, some
plaintext. Completion then marks the object by its majority and GetObject
fails decrypting the rest. Parts staged under .uploads now skip the
bucket-default lookup; the decision snapshotted into the upload entry at
CreateMultipartUpload is replayed through request headers instead, which
matches AWS (default encryption applies at initiation).
* s3: mark multipart part writes via request context, not the staged path
Inferring "is a part" from the .uploads file path let an ordinary PutObject
named .uploads/x.part skip bucket default encryption entirely. The part
decision now travels on the request context from the two callers that
stage parts (UploadPart, UploadPartCopy); a client-chosen key cannot reach
it.
S3DisableContentMD5Validation skips the request header that DeleteObjects
requires, so AWS rejects every directory removal and the filer remote sync
offset stalls. Issue the delete through DeleteObjectsRequest and re-enable
the flag on that request's config copy so the SDK's own checksum handler
adds the header without changing any other operation.
Generated with [Devin](https://devin.ai)
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* volume: replicate FetchAndWriteNeedle writes with the configured TLS scheme
The guarded upload path hardcoded an http:// URL and used a bare
*http.Client, so a peer fronted by https rejected the replica write and
read-through of a remote-evicted object failed whenever the volume had
replicas. Normalize the upload URL through the shared global client and
let the guarded transport borrow its https.client TLS configuration, the
same settings the regular replication path resolves.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* volume: keep the global HTTPS downgrade guard on replica uploads
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* volume: cover TLS replicate uploads and the redirect downgrade guard
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* volume: pin the downgrade guard against a dialable plaintext target
The previous redirect pointed at 169.254.169.254, which the guarded dialer
already refuses, so the test passed even with CheckRedirect removed.
Redirect to a live plaintext server on a dialable peer address and assert
it never receives the resent body.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* volume-rs: keep the HTTPS downgrade redirect guard on outbound writes
Mirrors the Go peer upload client: the shared outgoing HTTP client now stops redirects that would downgrade an https request to plaintext, so replicate uploads to TLS peers cannot leak data or auth headers via a 307 to http.
* volume-rs: fail rejected redirects instead of returning the 30x response
Go's rejectHttpsDowngrade uses ErrUseLastResponse, which surfaces to the
caller as a non-2xx status it rejects; attempt.stop() returned the redirect
response as Ok, so a refused downgrade could masquerade as a completed write
to any caller that does not check the status.
* volume-rs: fail replica writes on a non-2xx status
fetch_and_write_needle discarded the replica response without checking its
status, so a peer rejection — or a redirect the client refused to follow —
was reported back as a successful copy while no replica was written. Go's
doUploadData fails on any non-success status.
---------
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* vector: allow restricting ANN search to index segments
Query gains an optional index_segments list; when set and the index
path is used, the scanner restricts the vector index search to those
segments via with_index_segments. This is the seam the Enterprise
planner uses to fan one query out across per-segment workers; OSS
keeps passing None.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* vector: enumerate and partition index segments for fanout
segments::index_segments lists the physical segment UUIDs covering the
vector field at the pinned version; partition() assigns them
round-robin to workers. The parity test proves two workers scanning
disjoint shares merge back to the full-dataset top-K — the P3 exit
gate, in-process.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* vector: let a query carry fragment ids for the unindexed sweep
Lance only applies the index-segment restriction to indexed rows —
unindexed fragments outside the selected segments are skipped. For a
partitioned query to stay correct, the planner needs to hand the
unindexed sweep to a worker too, so Query gains a fragments list that
the scanner resolves to manifest fragments via with_fragments. A task
carrying both searches its segments via the index and its fragments
with flat KNN in the same scan.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* vector: empty scope returns no hits; cover the sweep in tests
Some([]) fragments or index_segments fell through to an unrestricted
scan — a worker assigned nothing would have searched everything.
empty_scope short-circuits those queries, and the segment restriction
only calls with_index_segments on a non-empty list.
segments::segment_fragments also returns each segment's live fragment
bitmap, and the parity test now leaves a fragment unindexed so the
worker carrying it exercises the flat sweep — the contract the
Enterprise planner relies on.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* vector: scope the scan to listed ∪ segment-covered fragments
Lance intersects with_fragments with the selected index segments, so a
worker given its segment share plus the unindexed sweep lost the share
entirely — only the listed fragments were scanned. Expand the fragment
list with the live coverage of the selected segments. The list is now
also applied when it resolves empty: stale ids or an empty assignment
must scan nothing, not fall through to an unrestricted scan. The merge
test now leaves a fragment unindexed so the combined path is exercised.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* vector: partition assigns nothing when there are no workers
n.max(1) returned every segment in one phantom partition for a
nonexistent worker, silently dropping coverage.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* webdav: refuse MKCOL and PUT under a missing folder with 409
RFC 4918 9.3 and 9.7.1 say a MKCOL or PUT whose parent collection does
not exist must fail with 409 Conflict. The filer creates missing parents,
so both succeeded and made the intermediate folders. Check the parent in
Mkdir and in OpenFile with O_CREATE and return os.ErrNotExist, which
x/net/webdav turns into 409.
* webdav: check the PUT parent before an overwrite deletes the target
OpenFile deleted an existing destination before the new parent lookup,
so a PUT whose parent check failed left the old file gone.
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* webdav: report a missing rename source as not-exist
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* webdav: refuse MOVE to a missing destination folder
AtomicRenameEntry grows any missing parents itself, so a MOVE to
/missing/file created /missing; check the destination parent first.
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* webdav: answer 409 for COPY and MOVE under a missing collection
The webdav handler reports a failed directory Mkdir or Rename as 403,
so the destination parent is checked at the request layer instead.
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* webdav: defer foreign-host MOVE and COPY destinations to the 502 handler
* webdav: only check the destination parent for absolute paths
* webdav: cover foreign-host and relative MOVE destinations in the guard test
---------
Co-authored-by: user.email <chrislusf@users.noreply.github.com>
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
checkAccessByOwnership compared the caller's account with the bucket
owner even with no identities configured, so Get/Put/Delete
BucketOwnershipControls answered AccessDenied to every request. With
auth off there is no owner to compare against, as the other auth-off
checks already assume.
The pinned dev snapshot (v1.85.0-dev.0.20260915183914) predates
grpc/grpc-go#9563, which fixes the control-buffer throttle deadlock
(grpc/grpc-go#9330) that wedges weed mini under concurrent S3 writes
over unix sockets.
Verified with the issue's standalone repro (one ClientConn over a unix
socket, concurrent unary RPCs): the old snapshot deadlocks at c=512
after 743 RPCs; the new snapshot completes 3.06M RPCs/20s at c=512 and
1.97M/15s at c=2048 with no stall -- above the v1.82.0 pre-regression
baseline of ~112k ops/s.
* topology: keep the oversized verdict after the location list is gone
The per-replica oversized marks live in the volume's location list, which
is deleted when its last location leaves and recreated flagless on the
next registration. Between that recreation and the next heartbeat that
re-measures the volume, ensureCorrectWritables saw no mark and
re-admitted an oversized volume to the writable list; two other paths
(quarantine release, mark-writable) never consulted the marks at all.
VolumeLayout now remembers the oversized verdict per vid, written only
by heartbeat sizes and forgotten only when a report measures the volume
under the limit. The writable admission gate itself refuses a marked
volume, and the vacuum commit path drops a verdict its live size check
has just disproved.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* topology: mark oversized at registration when the report carries a size
RegisterVolumeLayout ran EnsureCorrectWritables while the recreated
location list still held no per-replica marks. When the registering
report knows the volume's size, refresh the oversized state before the
writable check instead of waiting for the next heartbeat pass.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* topology: test that oversized volumes stay out of writables across re-registration
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* topology: remember the oversized verdict per replica
A shared per-volume verdict was cleared by any single replica's
under-limit report, so a still-oversized replica whose location entry
had died could re-register from a size-less report and put the volume
back on the writable list. Keep one verdict per reporting replica: an
under-limit report clears only its own, and a replica that never
returns cannot pin its volume out of writables forever because the
verdict ages out. The vacuum commit now feeds the node's fresh answer
through the same per-replica bookkeeping instead of erasing the whole
verdict.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* topology: count a volume at the size limit as full on vacuum commit
The layout's oversized test is >= the limit, but the vacuum commit and
status checks used >, so a replica sitting exactly at the limit passed
as not-full and the commit path could re-admit it.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* topology: cover per-replica oversized verdicts in tests
One replica shrinking must not clear a still-oversized replica's
remembered verdict; cover that plus a size-less re-registration before
the second replica is re-measured. The vacuum-commit case now shrinks
the stored record first, since the commit re-derives the mark from the
node's own answer.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* topology: judge the vacuum commit by the size it just measured
SetVolumeAvailable rebuilt the oversized verdict from the cached
VolumeInfo, which still carries the pre-compaction size until the next
heartbeat. A commit that shrank a volume under the limit re-marked the
replica oversized and never restored writes. The commit path already
asks every replica for its fresh size; pass it through per replica and
record it on the mark, the verdict, and the size tracking seed.
* topology: hold an expired oversized verdict while its replica is back
The TTL forgives a replica that never returns, but a remount announces
the volume without a size and registration skips the oversized check.
Deleting the verdict on expiry then re-admitted a still-oversized
volume until the next size-bearing heartbeat. Keep counting an expired
verdict while its replica sits in the live location list, and keep
sweeping verdicts whose replicas stayed gone.
---------
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: chrislusf <chrislusf@gmail.com>
calculateClusterCapacity multiplies advertised volume slots by the
volume size limit. Slot counts are an admission bound rather than a
capacity measurement — an explicit -max can be set higher than the
volumes a disk actually fills — so a small disk can report more total
and available capacity than it physically has. Bound both numbers by
the per-disk filesystem counters the volume server already reports in
its heartbeat, falling back to the slot estimate only when they are
absent.
Generated with [Devin](https://devin.ai)
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* mount: drain OFD locks when a file description is released
The kernel scopes OFD fcntl locks to the open file description but
reports no lock_owner for them at RELEASE — it only sends one for flock
holders — so an OFD write lock taken on a file handle outlived the
holder and blocked every later open of that file (e.g. etcd failing to
reopen its WAL after a restart). The mount now records the owner of each
granted non-flock lock under its file handle and releases those owners
on RELEASE, which also covers any POSIX locks left over when FLUSH could
not run.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* mount: give every open its own fh value over the shared handle
AcquireFileHandle reuses one FileHandle per inode, so every open reported
the same fh to the kernel. A RELEASE for one description could then not
be told apart from a sibling still holding the inode open — draining
lock owners recorded under the shared fh dropped locks belonging to
descriptions that stay open.
Each open now registers its own fh value in fh2inode, all resolving to
the shared FileHandle, and RELEASE removes exactly the id that is
closing. This also stops a duplicated RELEASE on one id from
double-decrementing the open count.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* mount: forget flushed lock owners so a later release cannot re-drop locks
POSIX lock owners are process-scoped: Flush releases every lock the
process holds on the inode, but the owner stayed recorded under each
open's fh for the Release-time drain. If the process took a fresh lock
through another open before an old description finished closing, the
stale record made that RELEASE drop the new lock too.
Flush now drops the released owner from every recorded open of the
inode, so the Release drain only ever fires for owners whose locks were
never flushed.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* mount: keep close-time lock owner records when the release fails
A routed POSIX owner release that errors keeps the flush hint for a
retry, but Flush also dropped the owner from every open's release
records unconditionally — so the final Release skipped the retry and
the owner filer's lock was stranded until lease reaping. Only forget
the owner once its locks were actually released.
---------
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* cluster: bound each DistributedLock attempt with a timeout
An unreachable lock owner blocked the attempt on the connection timeout.
Bound it so the write fails fast; a timed-out attempt keeps no renew token.
* cluster: keep renewToken across failed lock attempts
A request may have renewed the lease server-side before its response was
lost, so the last token can still be valid on the next attempt. Clearing
it on any transport failure strands a live lease until its TTL expires.
A definitive rejection still clears the token through the response's
empty RenewToken, and falling back to the seed filer no longer abandons
the lease either.
* cluster: bound each lock attempt by the remaining route window
The 15s unavailability window started only after the first failure, so
three 10s attempts plus sleeps outlasted it: the third failure selected
the seed fallback at the same moment the loop gave up, and a reachable
seed route was never tried.
Start the clock before the first attempt and cap each attempt at what
the window has left. When the window is spent on a non-seed target,
fall back to the seed filer — it can forward to the real owner — and
give that route its own window, so the fallback is always exercised
before acquisition fails.
* cluster: bound DistributedUnlock by the lock RPC timeout
The unlock used context.Background(), so Stop against an unreachable
lock server could wait on the TCP connect timeout far past the renewal
goroutine's bounded shutdown wait. A timed-out unlock is harmless: the
lease expires via its TTL regardless.
* dlm: replay the current token on a lost renewal's retry
A renewal rotates the token server-side before the response is sent. When
that response is lost the client only knows the superseded token: retries
mismatched, the token was cleared, and the holder conflicted with its own
lease until it expired. The lock now remembers the superseded token, so a
retry from the same owner replays the current token, and a superseded
token can still release its own lock.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* cluster: give the seed lock fallback one fresh window per acquisition
The seed can redirect back to a dead primary via LockHostMovedTo, so
refilling the retry window on every return to the seed made the
primary->seed->primary cycle unbounded. The first seed fallback keeps
its fresh window; later returns to the seed keep the running deadline.
* cluster: advance the lock seq when a lost-renewal replay extends the lease
A retry of a renewal whose response was lost updates ExpiredAtNs in place
but returned the existing Seq, so the replicated extension was rejected
by InsertBackupLock as a stale mutation. A promoted backup would then
grant the lock when the old expiry passed even though the holder still
believes the lease is renewed. Bump Seq on the replay so the backup
accepts the new expiry.
* cluster: carry the superseded lock token through replication and transfer
A renewal rotates the lock token and keeps the superseded one in
PrevToken so a holder whose renewal response was lost can still recover.
That recovery broke as soon as the lock moved: neither the replicated
backup copy nor a transferred lock carried PrevToken, so a new primary
rejected retries of the old token until the lease expired.
Add prev_renew_token to the Lock and ReplicateLockRequest messages and
thread it through ReplicateLock, TransferLocks, InsertBackupLock and
InsertLock.
* cluster: count clean lock redirects against the retry window
A bare LockHostMovedTo response (no error, no token) returns nil from
attemptToLockWithin, and retryUntilLocked reset the retry window on every
nil result. Two ring-disagreeing filers forwarding to each other therefore
bounced the client in an unbounded loop; the spent-window exit could also
return a nil error with no token, handing callers an unacquired lock.
* cluster: do not mark a lock held or hot-loop on tokenless attempts
A bare LockHostMovedTo redirect returns nil with no renewToken; the
attempt previously flipped isLocked to 1 anyway and retried immediately.
Gate the flag on a granted token and pace the retry like other outcomes.
* cluster: keep renewToken across bare lock redirects
A LockHostMovedTo response carries no token and no error; overwriting
renewToken with it dropped a held token, so the next renewal re-acquired
fresh and could contend against its own still-live lease until TTL.
* cluster: poll for replicated backup state in lock tests
Replication runs in a goroutine, so a fixed sleep followed by an
immediate backup read is racy. Wait for the expected backup state
with require.Eventually instead.
* cluster: mark a lock held only on a grant verdict
A bare LockHostMovedTo redirect with a saved token left isLocked set
without the redirected filer granting anything, and retryUntilLocked
could exit on a stale token with no attempt made. doLock now reports
whether the response was a grant, and both the retry loop and isLocked
follow that verdict.
---------
Co-authored-by: chrislusf <chrislusf@users.noreply.github.com>
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* vector: address remaining s3vectors review findings
Catalog and permission model:
- Vector buckets are declared Lance table buckets end to end: create sets
format=lance, get/list/delete reject or skip ICEBERG buckets, and the
format lookup runs on an internal trusted catalog so callers need no
GetTableBucket grant.
- The implicit "vector" index namespace is created through the same
internal manager, so a CreateTable grant alone authorizes CreateIndex;
typed S3TablesError codes replace string matching on err.Error().
- CreateIndex pre-checks existence after the authorize, tags the record
with a create nonce, and rolls back on engine failure under
context.WithoutCancel with a versionToken CAS, so a retry never deletes
a table it did not create.
- DeleteIndex drops the catalog record directly; DropIndex is
unimplemented on the shared VectorService.
- DeleteVectorBucket removes the empty internal namespace on
BucketNotEmpty and retries once.
- PutVectors/DeleteVectors require the update grant, and engine gRPC
errors map to typed API errors instead of generic 500s.
API correctness:
- CreateVectorBucket/CreateIndex return their resource ARNs.
- ARNs reject empty bucket segments; vectorBucketArn must be an ARN.
- ListIndexes returns summaries built only from catalog fields.
- ListVectors honors maxResults/nextToken inside a segment via
"seg:<pageStart>:<nextIndex>" resume tokens and validates
segmentCount/segmentIndex.
- QueryVectors requires positive topK; PutVectors rejects empty keys.
- CreateVectorBucket rejects a supplied encryptionConfiguration rather
than silently dropping it.
- Metadata filters emit Lance's real JSONB UDFs: json_get_string/
json_get_float/json_get_bool picked by operand type, json_exists with a
bracketed JSONPath, and per-member accessors for $in/$nin.
Engine:
- weed-vector-server stores the metadata column as Lance JSONB
(lance.json extension) so the json_* UDFs apply; reads accept legacy
Utf8 JSON columns.
Vectors port TLS:
- The TLS config is built before binding listeners, loads -cacert.file
into ClientCAs when client-cert verification is on, fails startup on an
empty/invalid CA file, and closes the gRPC conn on every failure path.
* s3vectors: authorize index create before writing the implicit namespace
A denied caller could still leave a persistent namespace behind because
ensureIndexNamespace wrote through the trusted catalog before the
caller's create grant ran. AuthorizeCreateTable now tolerates the
not-yet-declared implicit namespace (bucket policy alone decides) and
runs before any write; real table creates keep requiring the namespace.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* s3vectors: keep engine error details out of the API response
Unavailable now maps to 503 ServiceUnavailableException and unclassified
codes return a generic message; the real gRPC error stays in the log.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* s3vectors: exact-type metadata filters + NULL-safe $nin
The jsonb extractors convert numeric-looking strings through
json_get_float and 'true'/'false' through json_get_bool, so a stored '3'
could satisfy a filter for the number 3. Numeric and boolean comparisons
now also require the stored value not be a string, and each $in/$nin
group coalesces NULL so NOT cannot invert NULL into NULL.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* s3: validate the vectors TLS key pair before binding
The reloading certificate provider reads the files asynchronously, so a
missing or mismatched pair would only surface as failed handshakes on an
announced HTTPS port. Load it once up front.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* s3tables: make a token-checked table delete atomic
The version token was read in one filer call and enforced in another, so
a delete+recreate slipping between them let the delete erase the
replacement. When a token is supplied the name path now carries the
observed mtime as an if_not_modified_after precondition (evaluated under
the path lock), and a decoupled delete strips the catalog marker under a
compare of the captured attributes before touching the data path.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* s3vectors: compare filter operands against the stored JSON type
jsonb's strict converters make json_get_string return text for numbers
and booleans too, so a get_string IS NULL guard is never true for scalar
metadata — numeric and boolean comparisons silently matched nothing.
Numbers and booleans now compare through TRY_CAST of the serialized
json_extract text, which yields NULL on any other stored type; string
comparisons stay on json_get_string behind a CASE guard so the extractor
only runs on rows storing a JSON string.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* s3tables: allocate implicit namespace metadata before use
A first table create in an undeclared namespace took the
allowMissingNamespace branch with a nil record and panicked on the
owner assignment.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* s3tables: read the delete target's metadata and snapshot in one lookup
Two separate lookups let a delete+recreate in between validate the
caller token against the old record while the conditional delete aimed
at the replacement.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* s3tables: assert the captured record before a conditional table delete
An xattr update keeps Mtime, so the IfNotModifiedAfterNs guard alone
could not see a metadata write that had already rotated the version
token, and a recreate preserving the mtime could slip the mtime check.
The delete now first proves the entry still carries the captured
attributes under the filer's path lock, then keeps the mtime
precondition.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* s3vectors: match only JSON boolean literals in metadata filters
* filer,s3tables: delete versioned tables under one attribute precondition
* s3vectors: compare serialized JSON for string filter operands
json_get_string converts numeric-looking strings and booleans, so it
cannot serve exact-typed string comparisons even behind a guard. String
operands now compare the serialized json_extract text directly, which is
exact-typed and never errors; a Rust test pins the semantics against a
mixed-type dataset.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* s3vectors: compare boolean filter operands as serialized text
Arrow's boolean cast accepts '1' and '0', so a TRY_CAST column let a
stored number satisfy a boolean filter. Booleans now compare the
serialized json_extract text like strings do — a JSON boolean only ever
serializes as 'true' or 'false'.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* s3tables: claim a catalog name's markers in one compare-checked write
A create wrote its markers attribute-by-attribute after checking the
name was free. A delete's restore landing in the check-to-write gap put
the old markers back first, and the create then overwrote only the ones
it sets — leaving the old tags, policy, and maintenance settings on the
new table. CreateTable, RegisterTable, and CreateView now write all
their markers in a single update that refuses to run while any catalog
marker is present, so the second writer loses with a conflict instead
of inheriting them.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* s3tables: test the attribute-preconditioned table delete and name claim
Adds a BeforeDelete hook plus ExpectedExtended evaluation to the
in-memory filer so tests can land a competing write in a handler's
check-to-delete window, and covers two losing races: a token delete
whose record changed since the read, and a create colliding with a
delete's marker restore.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* s3tables: reserve a catalog record under one compare-checked claim
claimCatalogRecord writes a deleting marker onto the record while it
still holds exactly the caller's captured attributes, so a delete or
rename can hold the record through a multi-step operation and a losing
writer backs out before any data or destination entry is written. The
claim key joins the markers a name-claim asserts absent, so the name
cannot be reused while the reservation stands.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* s3tables: claim the record before purging or publishing past it
A token delete of a decoupled table purged the data first and checked
the marker afterwards, so a rename that finished in between left the
moved record pointing at purged data while the delete reported a
conflict. The delete now claims the record under the captured
attributes before any data is removed and releases the claim if the
purge fails, keeping the record intact for a retry.
The rename claims the source before publishing the destination, so a
losing rename backs out while no destination entry exists instead of
leaving one behind pointing at data a delete went on to remove.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* s3tables: test the record claim across delete and rename races
A rename completing in the token delete's check-to-claim window turns
the delete into a conflict with the data intact, and a delete's claim
on the source turns the rename into a conflict before the destination
entry is published.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* vector: version-lease helpers + maintenance safety tests
Pinned queries must survive Lance maintenance. maintenance::pin tags the
current dataset version so cleanup refuses to reclaim it; release drops
the tag. Tests prove the P2 gate: a pinned query returns identical hits
across compact_files, a tagged version survives cleanup_old_versions
(error_if_tagged protects it, the non-strict run skips it), and a
missing data file fails the query instead of returning partial hits.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* vector: lease the query's dataset version in the VectorService
query_vectors pins the resolved version with a query tag for the
duration of the RPC and releases it after, so concurrent maintenance
cleanup cannot reclaim the files out from under an in-flight read.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* vector: release the version lease on every query exit path
The lease tag was only released on query_vectors' success path — get_vectors
never released and query_vectors leaked its tag on early returns. pin now
returns a guard whose drop deletes the tag, so every path frees the version.
* vector: unique lease tag names + reap abandoned leases
Nanos-since-epoch tag names can collide across concurrent readers and
processes, so each lease now carries a unique suffix. pin() also drops
query tags older than an hour, since a killed or cancelled request
leaves its lease behind and would otherwise block cleanup forever.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* vector: keep live lease tags + throttle abandoned-lease scans
An expired tag is only reaped once no live lease in this process owns
it, so a query running past LEASE_TTL keeps its pin. Tag scans run at
most once per 60s per dataset instead of on every request.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* vector: answer read-only queries without a version lease
A namespace may hand the server credentials that can read a table but
not update it, so the tag write in pin() fails and the whole query
errors out. try_pin() degrades a denied tag write to an unleased query
instead of failing the request.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* vector: fail a lease whose pinned version is already gone
Cleanup lists tags once at the start, so a tag created mid-cleanup does
not protect the version it names. After the tag lands, pin now verifies
the version still exists and drops the lease when it does not, failing
the read at pin time instead of mid-query.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
test: assert the resumed segment token
The second listSegmentVectors result was never checked (staticcheck
SA4006); assert it resumes scanning at row 19 after matches 10..18.
* filer.remote.gateway: skip uploading a dropped local copy on update
The bucketed update path uploaded the chunkless entry whenever the update
was not metadata-only. remote.uncache, eviction, and remote.meta.sync drop
a synced entry's chunks and clear its sync stamp, which reads as dirty
when the remote mtime shares the entry's second; the upload then replaced
the object that is the entry's only content with zero bytes. Skip the
upload the same way the mounted-directory sync does.
* filer.remote.gateway: copy a remote-only entry's object on rename
A rename deleted the source object and uploaded the chunkless entry,
replacing the entry's only content with zero bytes — the same loss
remote.uncache caused on plain updates. Complete the rename on the
remote the way the mounted-directory sync does: copy the source object
over (or take the destination when it already describes the entry),
restamp so reads stop resolving through the pending-source marker, and
remove the source only while it is still the object the stamp describes.
A remote-only entry renamed out of the gateway's coverage keeps its
object, and the plain rename's old-key delete only removes the object
the event's stamp still describes.
* filer.remote.gateway: scope the sync-stamp skip to same-path updates
shouldSendToRemote reads the entry's sync stamp, which still covers a
rename's unchanged mtime, so a synced or pending-stamped rename event was
skipped before its copy or upload could run. Only updates in place may
skip on the stamp.
* filer.remote.gateway: keep the source a pending rename moved on from
A rename event whose pending destination has since been renamed again
still names the source a later event reads through its own marker;
deleting the old key here loses the only copy. Mirror the remote.sync
check: leave the source when the event's entry carries the pending-source
marker and no entry holds the destination any more.
* filer.remote.gateway: release a rename's pending source once complete
A rename whose old entry read through a pending-source marker leaves that
earlier source behind: the destination is released by its own key, never
the marker's. Mirror releaseRenamedSource — once the renamed entry no
longer reads through a marker, delete the pending source guarded by the
old entry's stamp, keeping it while a remote-only entry holds its path.
* filer.remote.gateway: finalize a complete rename before failing its source
A rename source outside the gateway's coverage errored before the
destination was checked, and the recorded failure held the event offset.
Stat the destination first: when it already holds the content the entry
describes, refresh the stamp and leave the unmanaged source untouched;
fail only when nothing holds the content.
* filer.remote.sync: document the source-of-truth protocol both processors share
---------
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test/vectors: give the vector server static storage credentials
The e2e harness ran weed-vector-server without credentials, so lance's
AWS object store fell back to IMDS and CreateIndex died on the runner's
411 from the token endpoint. Point it at the test credentials the s3
gateway now reads from s3.json.
* vector: take the list item field from the schema when batching vectors
Lance stores the fixed-size list item as nullable even when the write
schema declared it non-null, so PutVectors built batches that no longer
matched: expected FixedSizeList(4 x Float32) but found FixedSizeList(4 x
non-null Float32).
* vector: store index metadata as lance jsonb so filters resolve
The metadata field was plain Utf8, but the filter functions lance's
planner registers (json_get_string, json_get_float, json_exists) take a
LargeBinary jsonb column, so QueryVectors with a metadata filter failed
to plan. Declare the field with the arrow.json extension — lance encodes
it to jsonb on write and back on read — and encode the batch's strings
when the stored column is LargeBinary. Also rename the filter's
json_get_str call to json_get_string, the name lance registers.
---------
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* vector: boto3 s3vectors conformance harness
test/vectors spins a mini cluster, an s3 gateway with -port.vectors and
-vectors.grpc, and a weed-vector-server, then exercises the AWS client
end to end: bucket and index lifecycle, put/get/list/delete vectors,
and a filtered query asserting distances and metadata.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* ci: run the s3vectors conformance subset
An e2e job in rust-vector-tests: builds weed and weed-vector-server,
then runs test/vectors against a mini cluster plus the vectors gateway.
The workflow now also watches vector.proto, its stubs, and test/vectors.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* ci: fix e2e protoc and codespell findings
The e2e job needed the same vendored-protoc step as the unit-test job —
lance's build scripts locate protoc themselves. Codespell flags the AWS
`$nin` operator literal and the `lits` variable: exempt `nin` in
.codespellrc like the existing word exemptions, rename the variable,
and fold the negation into the IN/NOT IN symbol directly.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* s3: classify catalog errors by code, not message text
* s3: drop the implicit vector namespace on vector-bucket delete
* test: point the vector engine at the gateway lance port
* test: bind the vectors gateway listeners to loopback
* test: make the filter and distance assertions load-bearing
* test: rebuild the harness binaries on every run
* ci: fix the vectors end-to-end job
* ci: run the vector tests on gateway and command changes
---------
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* s3: nil-check metadataConfiguration before reading its keys
* s3: honor maxResults/nextToken in segmented ListVectors
* s3: exclude non-Lance buckets from ListVectorBuckets
* s3: test segmented ListVectors paging token math
* ci: silence codespell on the "$nin" operator and rename lits
* vector: S3 Vectors API gateway over VectorService
weed/s3api/s3vectors serves the AWS S3 Vectors JSON API: bucket CRUD
maps to S3 Tables table buckets, index CRUD to Lance table records plus
the VectorService CreateIndex/DropIndex RPCs, and the data plane
(Put/Get/List/Delete/QueryVectors) calls the shared service. The AWS
filter document translates to a Lance SQL predicate over the metadata
JSON column; unsupported operators fail loudly rather than overmatch.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* vector: -port.vectors and -vectors.grpc on weed s3
Mirror the Lance server startup: the S3 Vectors API binds its own port
and dials the weed-vector-server gRPC address; both disabled by default.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* vector: resolve index ARNs, authorize index ops, fix catalog fields
- vector buckets declare FormatLance so a new bucket accepts index
tables instead of defaulting to ICEBERG.
- ListIndexes decodes the `tables` response field; `tableSummaries`
hid every index.
- resolveIndexARNs honors indexArn alone; handlers no longer treat
indexName as the only carrier.
- data-plane RPCs fetch the index record first, so callers authorize
against the index resource and a missing index 404s before touching
the engine; QueryVectors reports the index's configured metric.
- IndexArn added to the index-level request types.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* vector: serve the vectors port over TLS when configured
When -key.file/-cert.file are set, the vectors listener mirrors the
main API and serves TLS rather than a second plaintext endpoint.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* vector: harden the s3vectors gateway handlers
- $ne no longer shadows the operator: exclusion filters emit <>
instead of equality.
- a null element in a PutVectors list fails the request instead of
panicking.
- requests are capped at 32 MiB; distances return via *float32 so an
exact match still reports 0 when returnDistance is set.
- CreateIndex declares the `vector` namespace first and validates
dimension (1..=4096) before the catalog write; a failed
VectorService.CreateIndex rolls the table record back.
- non-filterable metadata keys persist as JSON rather than a
comma-joined string, and index records carry their createdAt.
- ListVectors honors segmentIndex/segmentCount by slicing the full
listing in the gateway.
- catalog errors map to AccessDeniedException/ConflictException instead
of a blanket ValidationException, and malformed ARNs 400 instead of
slicing.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* vector: /{Operation} routes and vectors-port TLS, conn cleanup
Path-dispatched clients (POST /QueryVectors) reach the same handlers as
X-Amz-Target calls. A failed listener bind now closes the vector gRPC
connection instead of leaving it open with the endpoint disabled.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* ci: retry the timing-sensitive s3 lifecycle expiration tests once
test_lifecycle_expiration and test_lifecyclev2_expiration assert exact
object counts after fixed sleeps. The lifecycle worker expires objects
via meta-log replay, and under the suite's write flood a shard's replay
can trail an object's PUT event past the test's window (observed on
master and PR runs, SQL-store job only). Split them out of the main
list and allow one retry; each attempt is independent so real
regressions still fail.
* ci: allow a second retry for s3 lifecycle expiration tests
* s3: list every bucket when auth is off
With no identities configured, every request may create, read and
write any bucket, but ListBuckets treated the caller as an
unauthenticated user and returned an empty list, so `aws s3 ls` showed
nothing. List all buckets in that mode, with prefix, max-buckets and
continuation tokens as before. Listing with auth on is unchanged.
* s3: cover the auth-on anonymous ListBuckets case
The no-auth listing change also moves the nil-identity gate; pin the
security-relevant direction that a request without an identity still
sees no buckets once auth enforcement is on.
* s3: verify ListBuckets continuation token reaches the next page
---------
Co-authored-by: user.email <chrislusf@users.noreply.github.com>
* vector: VectorService protobuf contract
The shared RPC boundary between the Go S3 gateway and the Rust vector
engine from the vector design doc: CreateIndex/DropIndex, Put/Get/
List/Delete/Query vectors. A Table reference carries the Lance table id
and an optional pinned version, matching the snapshot discipline queries
are required to keep.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* vector: generated Go stubs for vector.proto
protoc output via weed/pb/Makefile.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* vector: carry return_data through QueryVectors
S3 Vectors QueryVectors honours returnData, but the contract had no
way to ask for or carry vector data in a query response, so the
gateway would need a second GetVectors round-trip per query. Add
return_data to the request and data to QueryResult now, while the
contract is still new.
* vector: pin metadata numbers to double precision in the contract
Metadata arrives as JSON, where numbers carry double precision; say
so on the field so neither side promises integer fidelity past 2^53.
* vector: declare_table on the lance namespace client
CreateIndex needs a way to ask the catalog for a new table's location;
expose POST /v1/table/{id}/declare.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* vector: cargo fmt coverage.rs
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* vector: weed-vector-server, the VectorService gRPC
A tonic server over weed-vector-core: CreateIndex declares the table
through the namespace and writes an empty {key, vector, metadata}
dataset; PutVectors upserts by key (delete + append until merge_insert
is exercised); Get/List/Delete/Query route through the engine's
resolved table, honoring pinned versions and the prefilter default.
Metadata travels as a Utf8 JSON column, proto Struct on the wire.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* vector: stamp the metric in the schema, tighten put validation
CreateIndex now bounds dimension at the S3 Vectors 4096 maximum and
records the distance metric in the vector field's Arrow metadata, so a
later reader recovers the index's configured metric from the table
itself. batch_of checks every vector's data length individually before
flattening — equal totals could still let a short vector borrow floats
from its neighbor — and writes against the dataset's own schema.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* vector: upsert vectors in a single merge commit
PutVectors ran delete-then-append: a failed append left the old keys
deleted, and two concurrent puts could both append a replacement. Use
lance's merge_insert on the key column — one commit, no delete window,
conflicts fail the request instead of duplicating keys.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* vector: read the index's metric; fetch metadata in one take
QueryVectors hardcoded L2 — it now reads the metric the index was
created with from the vector field's metadata. return_metadata fetched
each hit's row one at a time; collect the row ids into a single
take_rows call and join metadata back on the hit key.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* vector: honor return_data in query results
QueryResult grew a data field but query_vectors never populated it.
Fetch the vector column in the same batched take_rows as metadata and
return each hit's stored embedding.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* vector: reject a malformed query as invalid_argument
A zero top_k silently became one result and a dim-mismatched query
vector surfaced as an internal scan error; reject both before querying.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* vector: reject a malformed list next_token
A token that failed to parse silently restarted the listing from
offset zero; surface it as invalid_argument instead.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* vector: answer an empty put without a commit and cover ragged input
An empty vectors list would still run a merge commit; return the
current version instead. Pin the ragged-vector check with a test.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* vector: keep non_filterable_metadata_keys on the table schema
create_index dropped the declared filter policy. Persist it as JSON on
the metadata field so the index carries which keys a filter may touch;
enforcing the list belongs to the gateway that translates the request
filter into Lance SQL.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* vector: reject duplicate keys in one put
A repeated key for a new row committed both copies, and for an existing
key the merge failure surfaced as internal. Reject duplicates up front
as invalid_argument.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* vector: report open failures as not_found or unavailable
A catalog 404 or a missing dataset/version is not a client mistake, and a
catalog outage is retryable: mapping every open error to invalid_argument
made both indistinguishable from a bad request.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* vector: page list_vectors when max_results is unset
A proto3 client that omits max_results sends 0, which used to limit the
scan to zero rows and return no token.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* vector: drop stray empty service.rs.tmp
* vector: rename datas map to data_map
codespell flags datas as a misspelling of data; use a distinct map name.
---------
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* vector: VectorService protobuf contract
The shared RPC boundary between the Go S3 gateway and the Rust vector
engine from the vector design doc: CreateIndex/DropIndex, Put/Get/
List/Delete/Query vectors. A Table reference carries the Lance table id
and an optional pinned version, matching the snapshot discipline queries
are required to keep.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* vector: generated Go stubs for vector.proto
protoc output via weed/pb/Makefile.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* vector: carry return_data through QueryVectors
S3 Vectors QueryVectors honours returnData, but the contract had no
way to ask for or carry vector data in a query response, so the
gateway would need a second GetVectors round-trip per query. Add
return_data to the request and data to QueryResult now, while the
contract is still new.
* vector: pin metadata numbers to double precision in the contract
Metadata arrives as JSON, where numbers carry double precision; say
so on the field so neither side promises integer fidelity past 2^53.
* vector: clarify the query filter is raw Lance SQL over one metadata column
The doc said 'metadata columns' but the table stores all metadata in a
single JSON column named metadata; the gateway translates S3 Vectors
filter expressions onto it.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* filer: shut down test filers' meta-log buffers, fixing the 32-bit OOM
Each log_buffer.NewLogBuffer preallocates five 8 MiB windows and spawns
loopFlush/loopInterval goroutines that keep them reachable. The
weed/filer tests built ~70 filers and aggregators that almost never
called ShutdownLogBuffer, pinning ~2.8 GB plus live churn, and the
linux/386 job ran out of its ~4 GiB address space inside
newSealedBuffers.
- MetaAggregator.Shutdown now retires MetaLogBuffer too: the "aggr"
buffer's loops previously outlived the aggregator.
- newTestFiler and newTestFilerWithStubStore register t.Cleanup
shutdowns; the one-off filer/aggregator literals defer theirs.
* filer: wait for the aggregated meta-log buffer to stop in Shutdown
ShutdownLogBuffer only signals the stop; loopFlush and its pooled windows
exit asynchronously. Match Filer.Shutdown's local-buffer sequence so the
aggregator returns with the buffer's memory already released.
* filer: cancel test filers' flush context before buffer shutdown
The stub-store helpers build real NewFiler instances, so the queued
shutdown flush retries appendToFile every ~737ms under flushCtx, which
nothing ever cancelled — the buffer and its pool stayed alive past each
test. Cancelling first lets the flush drop the window and WaitForShutdown
confirms loopFlush exited.
* filer: wait for test log buffers to drain after shutdown
The stub-flush buffers cannot hang, but without WaitForShutdown their
loopFlush goroutines release the pooled windows asynchronously — the
memory this PR means to free is still counted as live while the next
test allocates.
* vector: --vector takes hyphenated floats again
* vector: --vector-column/--key-column on the get subcommand
* vector: --vector-column/--key-column on describe and coverage
* vector: surface describe resolve errors when column flags are set
* vector: cover the restored CLI flags with parse tests
* filer: deliver meta-log events under their stream position, not their stamp
A mutation's event timestamp is stamped before the filer store write so
the entry and its event share a commit time, while the log-buffer append
happens after the write. A write delayed behind later-stamped mutations
keeps its early stamp in the payload while AddDataToBuffer bumps
LogEntry.TsNs to keep the stream monotone - so subscribers could
receive events whose payload TsNs arrives out of order.
Consumers treat the delivered TsNs as a stream position (cursors,
watermarks, replay windows). The s3 lifecycle daily-run drains events in
order and pins its per-shard cursor behind the first not-yet-due match;
an event delivered with a stale stamp after the cursor already passed it
is filtered out forever and its pending match silently never fires -
the flaky test_lifecyclev2_expiration signature of exactly one object of
a same-rule pair surviving, with errors=0 on every pass.
Normalize event.TsNs to logEntry.TsNs at delivery (and for the
skeleton-filtered heartbeat) so delivered timestamps are monotone by
construction. The commit-time stamp remains available on the entries
themselves via CommitTsNs.
* pb,filer: stamp metadata events with a mutation timestamp
TsNs on the wire now carries the delivery stream position, so the
mutation's commit stamp needs its own field. Add mutation_ts_ns to
SubscribeMetadataResponse and set it when the event is created.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* filer: carry the mutation stamp through event delivery
Delivery rewrites TsNs to the stream position; the payload stamp survives
in MutationTsNs, backfilled from TsNs for payloads written before the
field existed. The notification callback forwards it to the wire message.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* pb: normalize chunk-replay timestamps like eachLogEntryFn
ReadLogFileRefs bypasses the server decode path, so a delayed append
still arrived under its stale payload stamp and could hide behind a
subscriber cursor. Deliver the stream position and preserve the stamp in
MutationTsNs.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* filer: order replay and tombstones by the mutation stamp
Replay treated the delivered TsNs as the commit time, so a peer write
delayed in its own log could overwrite a newer local entry. Compare and
stamp with MutationTsNs, falling back to TsNs for older peers.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* remote.sync: carry the mutation stamp through reshaped renames
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* filer: test replay ordering on the mutation stamp
* filer: tombstone remote deletes by delivery position, order by commit stamp
* filer: keep the peer commit stamp when restamping aggregated events
* pb: keep the log cursor on the entry position past callback restamps
---------
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
CI installs the templ CLI with `@latest`, which now generates code
calling templruntime.NewAttributeExpression — a symbol added after the
v0.3.1070 runtime pinned in go.mod, breaking `make install` builds on
clean runners. Bump the runtime to v0.3.1100 and regenerate.
* filer: format ChunkView without its cipher key
readChunkSliceAt logs a failed chunk fetch with %+v, which printed the
per-chunk AES-256-GCM key into error-level logs on every failed read.
Give ChunkView a String() that omits CipherKey so %v and %+v stay safe.
* pb: scrub cipher keys when an entry must be logged whole
ScrubCipherKeys clones an Entry with chunk CipherKey fields cleared, so
a warning that needs the full entry still cannot leak the keys.
* log entry identifiers instead of whole metadata messages
Several log lines formatted whole protobuf messages — SubscribeMetadataResponse,
EventNotification, Entry-carrying requests — whose chunks include cipher_key
under -encryptVolumeData, so the keys escaped into logs (some at default
verbosity). Log the directory, entry name and event timestamp instead; where a
chunk list was printed, print file ids via FileIdsString.
* log queue: unwrap SubscribeMetadataResponse and never print raw protos
The fallback printed any message with %+v, so a SubscribeMetadataResponse
carrying entries would still put cipher keys in the log. Unwrap its event
notifications (batched events included) and reduce unknown messages to
their type name.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* remote sync logs: name the event's real path
Rename targets logged under resp.Directory while the entry lives under
NewParentPath; route them through MetadataEventTargetDirectory. Sync
failure logs printed only the directory, so same-dir failures shared an
identifier — log the source and destination paths instead.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* log queue: log the entry's destination path on moves
A cross-directory rename printed "file" -> "file" under the source
path, hiding where the entry landed. Use NewParentPath so the log
shows the destination directory next to the new name.
* admin: install the go.mod-pinned templ, not @latest
install-templ fetched templ@latest, which now emits
templruntime.NewAttributeExpression - a symbol absent from the
runtime pinned in go.mod - so every fresh 'make' regenerated
uncompilable _templ.go files. Install the go.mod version and
reinstall when the binary on PATH reports a different one.
* admin: drop stray Makefile change belonging to the templ pin PR
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* notification: join the move destination path in log queue output
A NewParentPath of "/" printed //name for moves into the root;
path.Join keeps a single leading slash.
* log queue: skip a nil event notification instead of logging an empty move
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* pb: tolerate nil chunk entries while scrubbing cipher keys for logs
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* subscription and backup failure logs name the event's file
The process/marshal failure logs printed only the event directory and
timestamp, so a failed sync left operators guessing which file failed.
Use MetadataEventSourceFullPath/MetadataEventTargetFullPath so each
failure names the source and target paths without printing the entries.
* replication: identify entry-less events without printing the proto
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* s3: scrub cipher keys from the bucket-metadata debug dump
buildBucketMetadata marshals the whole filer entry to JSON for a V(3)
log line; a chunked entry would emit each chunk's cipher_key base64.
Marshal the scrubbed copy instead.
---------
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* s3: keep transient object-entry errors out of ErrObjectNotFound
getObjectEntry wrapped every fetch error as ErrObjectNotFound, so a
transient filer failure (busy filer, ctx deadline, transport blip) was
indistinguishable from a real miss. LifecycleDelete resolves
ErrObjectNotFound permanently as NOOP_RESOLVED, so one flaky lookup at
dispatch time silently skipped the object's expiration - errors=0, no
retry, no log. This matches the flaky s3tests
test_lifecyclev2_expiration signature: exactly one object of a
same-rule pair surviving past its window.
Only collapse genuine misses (filer not-found, version/latest-version
sentinels); everything else propagates so the worker retries it as
RETRY_LATER instead of resolving it.
Also seed bucketConfigCache in the retention test server and teach the
fake filer to inject per-path lookup errors.
* s3: keep invalid version ids on the not-found path in getObjectEntry
A malformed versionId used to collapse into ErrObjectNotFound like every
other fetch error, so the handlers answered NoSuchKey. With transient
errors now propagated, an invalid id would surface as InternalError;
fold it back into the miss classification.
* filer_pb: keep Unavailable out of LookupEntry's not-found normalization
* s3: reject Unavailable in isObjectEntryMiss before message matching
* s3: keep transient fallback errors out of latest-version resolution
* filer_pb: limit LookupEntry's not-found text match to miss statuses
* s3: limit isFilerNotFound text matching to miss statuses
* vector: project _distance and _rowid explicitly
Lance warns that a knn projection without _distance will stop
auto-including it; spell both scoring columns out.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* vector: reference-comparison tests
The spike's exit gate: assert the engine's answers equal a bare
scan().nearest() on the same pinned version — over a fresh index, over
unindexed fragments, over a two-segment index after optimize_indices,
and after a delete. A version-pinned handle querying through
maintenance sees only its own version.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* vector: weed-vector spike CLI
describe / coverage / query / get against a namespace-registered table
or a plain dataset URI, so the spike runs against a real SeaweedFS Lance
table bucket and against a local fixture alike. Flag and env names
mirror weed-worker.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* ci: rust-vector-tests workflow
Build and test seaweed-vector in release on pushes and PRs touching it
or its path-dependency trees, mirroring rust-worker-tests.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* vector: derive Serialize for the spike CLI's JSON output
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* vector: --postfilter on the spike CLI
Track the Query field rename; the flag stays an explicit opt-out of
prefiltering for comparison runs.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* vector: assert the pinned query's answers
The maintenance test passed even with zero pinned hits: an empty result
satisfies all() trivially. Assert the hit count and compare the pinned
answers against the reference on the pinned handle.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* ci: vector tests also watch plugin.proto
weed-vector-core path-depends on weed-lance-worker, which compiles
weed/pb/plugin.proto through seaweed-worker-core; a proto-only change
would skip these checks and could break this workspace.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* vector: describe prints layout and coverage
The describe doc comment promised the vector layout and index coverage;
print both, and propagate count_rows errors instead of printing 0.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* vector: share the test dataset builder
The coverage tests need the same tempdir Lance datasets the query tests
build; move them into a cfg(test) helper module.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* vector: prefilter the metadata filter
prefilter=false post-filters the top-K rows, which can return fewer than
k results and never sees unindexed rows beyond the nearest cut. Expose
the choice so the S3 Vectors layer can honour AWS filter semantics.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* vector: index coverage introspection
coverage() unions each index's fragment bitmaps against the manifest's
fragment list to report which fragments a vector index covers, which it
has not reached, and how many rows the flat pass still has to scan.
Tests pin the behaviour the spike is for: rows appended after an index
build stay in the top-K, and a version-pinned dataset never sees the
next commit.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* vector: default filtered queries to prefilter
postfilter=false is now the normal path: the metadata filter cuts the
eligible rows before ranking, so a query returns k hits whenever k
eligible rows exist. Lance's post-filter remains reachable through
Query::postfilter for comparison runs.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* vector: intersect index bitmaps with the live manifest
A bitmap records fragments the index covered when built; a fragment the
manifest dropped afterwards stayed in the reported coverage.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* vector: keep bitmap-less indices in coverage
An index whose metadata has no fragment bitmap was dropped from the
report entirely, indistinguishable from no index at all. Count its
segments and mark its coverage unknown instead.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* vector: count live rows in unindexed coverage
physical_rows counts deleted rows and can be absent in the manifest;
sum FileFragment::count_rows for the unindexed fragments instead.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* vector: error on a null key in a selected key column
* vector: compare filtered ANN and flat hits as sets
* vector: coverage reports the checked-out version
* vector: project _distance and _rowid explicitly
---------
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: chrislusf <chris@seaweedfs.com>
* vector: scaffold the seaweed-vector workspace
Standalone cargo workspace for the vector query engine, mirroring the
seaweed-worker layout: own lockfile, same lint policy, path dependencies
across the two trees since there is no common root manifest.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* vector: resolve a table's vector layout
VectorTable::resolve finds the vector column (explicit name, else the
column named vector, else the sole fixed-size float list), the optional
key column, and the metadata columns a query can project.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* vector: flat and indexed top-k, key lookup
query::flat runs an exact scan (use_index(false)); query::ann lets Lance
split the query between index segments and a flat pass over fragments no
index covers yet, so rows written after the last index build stay
eligible. Both honour a metadata filter and return the raw _distance in
the query's metric.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* vector: reject a non-utf8 or non-identifier key column
An explicit --key-column naming a non-UTF8 field made hits() silently
report row ids as keys, and get() interpolated the name into a SQL
filter verbatim. Require Utf8 at resolve and in hits(), and require a
plain identifier before building the IN predicate.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* vector: fall back to row id on a null key
keys.value() on a null slot fabricates an empty-string key; a row with
no key reports its row id instead, matching the no-key-column case.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* vector: quote the key column in the IN filter
A Utf8 key column named e.g. document-key made the predicate parse as
subtraction, and an identifier interpolated unquoted is injectable.
Backtick-quote it and double its inner backticks; double quotes are
string literals in the filter dialect, not identifiers.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* vector: project only the columns a hit needs
The knn projection pulled every metadata column — including other
vector columns — although Hit returns none of them. Filters still
evaluate off the scan path, so restrict the output to the key and the
scoring columns.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* admin: install the go.mod-pinned templ, not @latest
install-templ fetched templ@latest, which now emits
templruntime.NewAttributeExpression - a symbol absent from the
runtime pinned in go.mod - so every fresh 'make' regenerated
uncompilable _templ.go files. Install the go.mod version and
reinstall when the binary on PATH reports a different one.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* admin: invoke templ from Go's install dir, not PATH
A templ binary earlier on PATH than GOBIN still shadowed the pinned
compiler after install-templ ran, so generate kept emitting code for
the wrong runtime. Resolve GOBIN (or GOPATH/bin) and call the pinned
binary by absolute path for install checks, generate, watch, and fmt.
* admin: regenerate templ output with the pinned v0.3.1070
The checked-in _templ.go files were stamped v0.3.1020, older than the
go.mod pin. Regenerate so the committed code matches the compiler the
runtime links against.
* admin: resolve templ under the first GOPATH entry
go install drops binaries in the first GOPATH element's bin, so a
multi-element GOPATH left TEMPL pointing at a path that does not exist.
Split on the OS list separator so Windows GOPATHs resolve too.
* admin: install pinned templ before fmt and pattern-rule generates
fmt and a direct %_templ.go rebuild invoked TEMPL without ensuring the
pinned binary exists; an order-only prerequisite keeps the check cheap
without forcing regeneration.
* admin: keep spaced GOPATH/templ paths intact in the Makefile
---------
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* master, volume, filer: start on a read-only mounted directory and serve existing data read-only
A read-only mount was treated as writable (permission bits only), so every
service failed on its first write: the master panicked writing raft conf.tmp,
the filer exited opening the leveldb LOCK, and the volume server skipped every
volume while still offering the directory as free capacity.
- util.IsReadOnlyMount: access(W_OK) == EROFS.
- volume: on a read-only -dir take the existing O_RDONLY/noWriteOrDelete load,
index .idx in memory when no .sdx exists, offer no free slots (same branches
as isDiskSpaceLow), skip the vid stamp there.
- master: on a read-only -mdir run raft from a private copy and refuse new
volume ids.
- filer: leveldb2/leveldb3 set their existing ReadOnly option.
* volume: skip read-only mounts when picking EC shard copy targets
A shard copy landing on a read-only mount fails at write time; offer
only writable locations like the other placement paths do.
* volume: mirror the read-only mount handling in the Rust volume server
The .dat open only fell back to read-only on EACCES, so a real read-only
mount (EROFS) still failed the load. The index side already degrades
once no_write_or_delete is set. Add is_read_only_mount via access(W_OK)
== EROFS, mark DiskLocation.read_only_mount, and mirror the Go checks:
no free slots, heartbeat max = used, no vid-stamp writes, no EC shard
copy targets, and a read-only index dir forces the volume read-only.
* volume: write vid stamps while only the index dir is read-only
* filer: skip the write-bit check when a leveldb store opens read-only
* filer: serve reads of buckets with no leveldb3 dir on a read-only store
* volume: derive a stable dir uuid when a read-only mount cannot store one
* master: keep the read-only raft copy across restarts so votes survive
* volume: index a missing .idx in memory on a read-only mount
* volume: release the .idx handle after an in-memory read-only load
* volume: mount EC shards read-only when their index dir is on a read-only mount
* volume: mark the index dir read-only before the tail repair opens .idx
* volume: write vid stamps while only the index dir is read-only (Rust)
* volume: derive a stable dir uuid and report no free slots on a read-only mount (Rust)
* volume: index a missing .idx in memory on a read-only mount (Rust)
* volume: refuse an EC copy whose shard owner is on a read-only disk
* master: key the read-only raft copy by the absolute -mdir
Two masters sharing the same relative -mdir text from different working
directories would have collided on one temp copy and shared raft state.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* volume: mix device and host into the read-only dir uuid
Deriving the uuid from the path alone let independent servers at the
same mount point report one uuid, which RegisterUuids rejects. Hash in
the device id and hostname so restarts stay stable but disks differ.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: vic6996-web <320442181+vic6996-web@users.noreply.github.com>
Co-authored-by: user.email <chrislusf@users.noreply.github.com>
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix 12074
* volume: shuffle a copy of cached replica locations for proxy reads
LookupVolumeId serves cached location slices shared between lookups; shuffling in place races with concurrent proxied reads of the same volume.
* seaweed-volume: forward inbound method and surface lookup failure in proxy reads
Mirror the Go handler: proxy_request used to always send GET, turning a client HEAD into a replica download, and a failed master lookup returned 404 like missing data. Forward the inbound method and answer 500 for lookup failures in proxy mode.
* volume: keep master lookup error detail out of the proxy error response
* volume: bound the proxied-request wait in the method-preservation test
* seaweed-volume: answer 404 for missing volumes in proxy-mode lookups
Mirror Go's LookupVolumeId: a per-volume error in the master's lookup
reply (e.g. a missing volume) means "no locations" and answers 404
downstream, not a lookup failure that proxy mode answers 500.
* volume: guard nil lookup result in proxy path
A master response that omits the requested vid leaves
LookupVolumeId returning a nil *LookupResult with a nil error;
dereferencing it panics the read handler.
* seaweed-volume: answered lookup errors are missing data, not failures
A master reply carrying a per-volume error with HTTP 404 (volume not
found) now yields empty locations like Go's LookupResult, so proxy mode
answers 404. Other failure statuses such as 503 during master warmup
remain lookup failures that proxy mode answers 500.
* seaweed-volume: drop hop-by-hop headers from replica proxy reads
Go's inbound request drops Host, Content-Length and Transfer-Encoding
into dedicated fields, so its forwarded header set never carries them.
hyper keeps them; forwarding a client's Content-Length with no body
stalls the replica's read, and forwarding its Host misnames the target.
* volume: bound the proxy master lookup to the request context
* volume: drop hop-by-hop and Connection-nominated headers from proxy reads
* seaweed-volume: drop Connection-nominated headers from proxy reads
* volume: skip the proxy lookup error when the inbound request is gone
* volume: drop proxy-specific headers from replica proxy reads
* seaweed-volume: drop proxy-specific headers from replica proxy reads
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* volume: drop hop-by-hop headers from replica proxy responses
* seaweed-volume: drop hop-by-hop headers from replica proxy responses
* seaweed-volume: a failed lookup status without an error body is still a failure
* volume: do not re-proxy an already-proxied read for a missing volume
A proxied request that reaches a server which still lacks the volume was
looked up and forwarded again, so a stale location answer could bounce
the read between two servers until the client gave up. The Rust volume
server's proxy_missing_volume already returns 404 for proxied=true;
mirror it here.
* seaweed-volume: replace stale proxied params in the forwarded query
Go builds the proxied URL with query.Set(reqIsProxied), which replaces
every proxied param the client sent. Appending after a client-supplied
proxied=false left the replica's first-value lookup reading the stale
marker, defeating the loop guard.
---------
Co-authored-by: chrislusf <chrislusf@users.noreply.github.com>
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* pb: checkpoint metadata-log replay progress per settled entry
A chunk-mode subscriber's resume cursor only advanced after the whole
LogFileRefs batch drained, so a stream dropped mid-replay resubscribed
from the original stale position and re-read the same log files while
live traffic kept appending - under a flapping filer the replay window
never shrank, and each reconnect re-listed and re-fetched the same
backlog.
ReadLogFileRefs now reports every entry the merge has settled -
delivered or filter-skipped - and FollowMetadata checkpoints StartTsNs
on it, so a reconnect resumes where the previous stream stopped and the
backlog drains across restarts. The peer aggregator wires the same
progress into its resubscribe/offset position.
* pb: emit a replay checkpoint only after the merge passes its timestamp
Checkpointing an entry's timestamp the moment it settles could strand a
same-timestamp entry still queued on another filer's stream: resuming is
exclusive, so a reconnect would skip the unsettled half of the tie. The
merge now emits a timestamp only once a later entry proves every tied
entry was delivered (and once at clean EOF).
* pb+filer: never regress the replay cursor below the settled checkpoint
The drain's final assignment took the last settled entry's own timestamp,
which can lag the log position progressFn just checkpointed - a skewed or
legacy entry rewinds option.StartTsNs (and the peer's lastTsNs), so the next
drain or a resubscribe re-reads the settled prefix. Guard both assignments
so the cursor only moves forward.
* pb: cover the resubscribe path of replay checkpoints
A drain that aborts mid-replay leaves StartTsNs at the last checkpoint
the merge passed, and the next subscribe resumes there instead of
re-reading the whole ref set. The test drives the real
makeSubscribeMetadataFunc path: refs, a mid-batch apply failure, then a
resubscribe whose SinceNs must carry the checkpoint.
* pb: emit the tail replay checkpoint only for a settled timestamp
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(volume): keep preallocated growth above -volume.minFreeSpace
When free space fell below -volume.minFreeSpace, every volume went
read-only and the master kept growing new volumes. With preallocation on,
each new volume reserved its full size, driving the disk further below the
floor (15 Oct incident: 66 empty volumes reserved about 1980 GiB).
Two causes, both in weed/storage/store.go:
- MaybeAdjustVolumeMax sized a disk's volume slots from raw free space, so
space inside the floor was counted as available. It now counts only free
space above the floor.
- addVolume trusted the once-a-minute low-disk flag. A preallocated volume
is now refused when the disk is low now, or when its reservation would
not fit above the floor. Volumes without preallocation, including merges
that copy existing data, are not affected.
addVolume runs under a store-level lock, so two concurrent creates cannot
both pass the room check against the same free space.
* volume: mirror headroom guard in the rust store
find_free_location only trusted the once-a-probe is_disk_space_low flag,
so a burst of preallocated creates could still run the disk under
-volume.minFreeSpace between probes, and maybe_adjust_volume_max sized
slots from raw free space. Adds MinFreeSpace::is_low/available_space to
match Go and applies both checks the same way. store.write() already
serializes add_volume, so no extra lock is needed.
* volume: read rust headroom tests from a fake disk
The floor arithmetic used a live free-space reading, so unrelated disk
activity could flip a test between pass and fail. Store.disk_stats_of is
now the same seam Go uses (diskStatusOf): the growth-loop and floor tests
run against a controlled disk, mirroring the Go suite one for one.
* volume: hold back create metadata from the headroom check
An exact-fit preallocation left free space exactly at the floor, so the
.vif and .idx writes that follow a create could dip under it and get the
location flagged read-only at the next probe. Reserve room for .vif on
the data filesystem, and for .idx growth only when dir_idx shares it.
* volume: mirror the metadata allowance in the rust store
Same rule as the Go side: charge .vif against the checked filesystem and
.idx only when dir_idx shares it, and grow the exact-fit test's fake
disk by the per-create cost.
* volume: skip the reservation check where create cannot preallocate
* volume: mirror the platform preallocation gate in the rust store
* volume.copy: count the preallocation reservation in the destination space check
The copy path picked a location with room for the copied .dat, then reserved
the master's full volume size limit on it. A volume smaller than the limit
could land on a disk whose free space above -volume.minFreeSpace fit the
copy but not the reservation, driving the disk under the floor the same way
the growth loop did.
Fetch the preallocation setting before choosing the location and require
room for the reservation itself, only where CreateVolumeFile actually
reserves blocks.
* seaweed-volume: store min-free-space-adjusted available space like Go
Go's DiskLocation.AvailableSpace holds free space above the
-volume.minFreeSpace floor; the Rust probe cached raw free instead, so the
VolumeCopy location check could accept a disk already inside the floor.
Store the same adjusted figure and cover it with a test.
* seaweed-volume: reserve the copied .dat like Go's CreateVolumeFile
VolumeCopy called set_len for the reservation, which only grows a sparse
file — no blocks are held, so a disk with room for the copied bytes but
not the full reservation only fails mid-copy. preallocate_file fallocates
KEEP_SIZE on Linux like Go, reserving the blocks up front; off Linux it
is a no-op, matching PREALLOCATE_SUPPORTED.
* seaweed-volume: charge a copy's destination pick for the .dat reservation
Go's VolumeCopy now sizes its free-location check against the master's
volume size limit when the platform preallocates, not just the copied
bytes. The Rust pick only measured the copied bytes, so a destination
could accept a copy whose reservation then dips under the floor. The
config fetch moves ahead of the pick (same order as Go) and feeds both
the space check and the reservation itself.
* volume.copy: charge the picked location's free space under volumeCreateLock
The copy path picked a destination off the once-a-minute AvailableSpace
probe and released the decision long before the .dat reservation landed.
Two copies could pass the same cached headroom and together drive the disk
under -volume.minFreeSpace.
ReserveFreeLocation serializes the pick under volumeCreateLock, verifies
the disk still has live room for the reservation, and charges it against
the location's cached free space until the next probe refreshes it.
* seaweed-volume: mirror the copy pick's live-check and headroom charge
Go's VolumeCopy now picks through Store.ReserveFreeLocation: live disk
stats under the store write lock, then the reservation charged against the
location's cached free space. The Rust plan_copy_destination gains the
same store.reserve_free_location so two copies cannot spend the same
headroom before the first .dat reservation lands.
* volume.copy: saturate the free-space charge against a live-space drop
The two statfs reads bracketing a pick can race a large external write; a
plain subtraction would underflow and leave the location looking roomy
until the next probe.
* volume.copy: release the headroom reservation when a copy does not land
ReserveFreeLocation charges the picked location's cached free space, but
a failed copy never returned it, so the headroom stayed booked until the
next disk-space probe and blocked retries. ReleaseFreeLocation refunds
the charge; the copy's cleanup defer now registers with the reservation
so every failure path (existing-replica delete, .note write, stream send)
releases it, including the post-transfer mount/validate failures covered
by the outer defer.
* seaweed-volume: mirror the copy headroom release on failed copies
release_free_location refunds reserve_free_location's charge. The copy
returns it when the plan's existing-replica delete fails, when the .note
write fails, and in cleanup_failed_copy, so a failed copy stops holding
the headroom until the next disk-space probe.
* volume.copy: keep pending copy headroom out of probe readings
ReleaseFreeLocation refunded the reservation into AvailableSpace even when
a CheckDiskSpace probe had already rebased the cache on live stats — the
probe's reading still shows the in-flight copy's bytes as free, so the
refund double-counted headroom and two copies could spend the same space,
driving the disk under -volume.minFreeSpace.
Track the outstanding charge in DiskLocation.ReservedSpace instead: probes
subtract it so they cannot re-spend it, a failed copy removes only its own
charge, and a landed copy commits it — the charge and the new usage cancel
out. hasRoomForNewVolume respects pending headroom too, so a preallocating
create cannot overrun an in-flight copy.
* seaweed-volume: mirror pending copy headroom across disk probes
reserved_space tracks the in-flight copy's charge separately from the
cached free space: check_disk_space subtracts it, release removes only the
failed copy's charge, a landed copy commits it, and has_room_for_new_volume
respects it — matching the Go side.
* volume.copy: settle a copy's headroom as its bytes land on disk
The pending charge held the copy's full reservation until commit, but
written and fallocated bytes already show up in statfs, so the live-space
check counted them twice. A slow copy could then block other copies and
volume growth despite free space above -volume.minFreeSpace.
Settle the charge instead: commit the bytes CreateVolumeFile actually
reserved (a failed fallocate leaves nothing committed), then settle each
file's write progress as it lands. The failure paths now release only the
unsettled remainder; allocatedSize is exported for the .dat's st_blocks.
* seaweed-volume: mirror copy headroom settling as bytes land
reserved_space keeps only the promised bytes statfs cannot see yet:
preallocate_dat settles whatever blocks the filesystem actually reserved
(0 when fallocate cannot), each copied file's progress settles the rest as
it lands, and a failed copy releases only the unsettled remainder —
matching the Go side.
* volume.copy: release a failed copy's headroom charge only once
Both defers fired on the same in-closure error: the inner released the
unsettled charge and the outer released it again, driving ReservedSpace
negative (an atomic.Uint64 underflow) and inflating AvailableSpace —
the location would then fail every later headroom check, or worse,
pass them with fabricated space. Zero the charge once it is released;
the Rust mirror already releases via swap(0) for the same reason.
* volume.copy: write into the preallocated .dat instead of staging over it
The .dat stream staged into .copying and renamed over the file
CreateVolumeFile had just fallocated, so the copy's transient footprint
was preallocate+dat — more than the reservation the pick just checked —
and the reservation itself was discarded at rename. Writing in place
reuses the fallocated extent, keeps the peak at the reserved maximum,
and leaves the copied volume preallocated like a created one.
* seaweed-volume: mirror the in-place write into the preallocated .dat
The .dat copy opened the just-fallocated file with truncate, freeing its
blocks at open and spending the headroom again as the stream re-grew it.
Append mode preserves them, matching the Go side.
---------
Co-authored-by: chrislusf <chrislusf@gmail.com>
Co-authored-by: user.email <chrislusf@users.noreply.github.com>
* fix: volume.fsck -volumeId purges live chunks referenced from outside the volume's bucket
-volumeId narrowed the filer read to <bucketsPath>/<collection> when
the selected volumes shared one bucketed collection (#9347). A filer
started with -collection, or an fs.configure rule, stores files from
any path in that collection, so the narrowed read misses their
references and reports live chunks as orphans; with
-reallyDeleteFromVolume they are deleted. #11794 already made
-collection read from the root for this reason. Do the same for
-volumeId: the flag keeps selecting which volumes are checked and
purged, and the filer is always read from "/".
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
* test: volume.fsck keeps chunks referenced outside the volume's bucket
---------
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
Co-authored-by: user.email <chrislusf@users.noreply.github.com>
* iceberg: fix CREATE TABLE rejecting a partition on the first column
Apache Spark number the columns of a new
table from 0 and leave assigning fresh IDs to the catalog, so a table
partitioned or sorted by its first column arrives with source-id 0. The
create-table request decoded its partition spec and write order into
iceberg-go's bound PartitionSpec and SortOrder, which require positive
source IDs, so every such CREATE TABLE, and CTAS through stage-create,
was refused with "Malformed request: Invalid request body".
Decode both as iceberg.UnboundPartitionSpec and table.UnboundSortOrder,
as iceberg-go documents for catalog create-table requests;
newTableMetadata already resolves the placeholders through the request
schema by name and assigns fresh IDs. A spec or order that does not
resolve against the schema is now answered with 400 instead of 500, and
the decode error message carries the underlying reason.
Tests check that Spark's request bodies resolve to the same metadata as
1-based placeholders, that the handler accepts them, that unresolvable
source IDs get 400, and add a Spark integration test that partitions by
the first column.
* iceberg: gofmt the create-table source-id tests
---------
Co-authored-by: user.email <chrislusf@users.noreply.github.com>
* filer: support nanosecond timestamps for conditional delete (#11339)
* shell: fs.verify prune carries the entry mtime in nanoseconds
IfNotModifiedAfter now resolves to sub-second precision on the filer,
so pass the looked-up mtime_ns along with mtime; the re-lookup check
catches same-second rewrites that share the second, and the test mock
mirrors the filer's seconds/nanoseconds cutoff.
* shell: volume.fsck guards empty-directory deletes with mtime nanoseconds
The looked-up mtime_ns joins the seconds in IfNotModifiedAfter so a
directory rewritten within the same second is not dropped; the stub
filer mirrors the filer's scale heuristic and the race fixtures use a
realistic mtime so the nanosecond branch engages.
* filer: cover the seconds guard and equality boundary
Test second-resolution cutoffs on both sides of the entry's second and
the exact-equality nanosecond guard that still deletes.
* filer: compare the nanosecond guard as a time, not UnixNano
UnixNano overflows for mtimes outside 1678-2262, letting a far-future
entry slip past the guard; time.After compares exactly at every epoch.
The seconds branch keeps Unix() so a same-second cutoff still deletes.
* filer_pb: add explicit if_not_modified_after_ns to DeleteEntryRequest
* filer: prefer explicit nanoseconds guard, keep if_not_modified_after as seconds
* shell: send seconds and nanoseconds delete cutoffs, skip unexpressible mtimes
* filer: merge conditional-delete guards into one nanosecond cutoff
DeleteEntryMetaAndData carried both the seconds and nanoseconds guards
even though the precedence between them is fixed, so every caller passed
a dead parameter. The gRPC handler now folds if_not_modified_after into
nanoseconds at the boundary, where (sec+1)*1e9-1 keeps the whole-second
cutoff exact, and the filer takes a single ifNotModifiedAfterNs.
* filer: hold the conditional-delete cutoff as a time, not int64 nanoseconds
A seconds-only guard at or past year 2262 saturated to MaxInt64
nanoseconds (~2262), so a delete whose cutoff matched a far-future mtime
was skipped while the request still reported success, and any entry
between 2262 and the real cutoff survived pruning. The merged cutoff is
now a time.Time, where a whole-second guard maps to the last nanosecond
of its second for any int64 second.
---------
Co-authored-by: user.email <chrislusf@users.noreply.github.com>
Co-authored-by: Chris Lu <chris.lu@gmail.com>
* s3: group ListMultipartUploads by delimiter into CommonPrefixes
The delimiter was echoed but ignored, so every upload was listed. Keys
that continue past the delimiter after the prefix are now listed once as
a common prefix, counted toward max-uploads like an upload.
* s3: bound tracked multipart common prefixes to the page
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* s3: omit upload id marker when a page ends on a common prefix
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: user.email <chrislusf@users.noreply.github.com>
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* webdav: copy a file onto a collection URL without losing the collection
An overwriting COPY of a file to dir/ removed dir, then OpenFile refused
the name ending in / and the request failed with 403, leaving neither.
Drop the trailing slash from the destination when the source is a file.
* webdav: cover file-over-collection COPY through the real handler
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* webdav: keep a root destination out of the slash trim
A file COPY or MOVE onto / would rewrite the Destination to an empty
path, turning the existing refusal into x/net's bad-gateway error; leave
the root untouched so RemoveAll refuses it as before.
---------
Co-authored-by: user.email <chrislusf@users.noreply.github.com>
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
The 24.04 bump broke build-rust-binaries: aws-lc-sys C objects compile against noble's glibc 2.39 headers, which redirect sscanf/strtol to the C23 __isoc23_* variants musl does not provide, failing the link.
Same fix as the enterprise build on seaweedfs/artifactory:
- arm64: keep aarch64-linux-gnu-gcc, link a tiny forwarding shim that
resolves __isoc23_sscanf/__isoc23_strtol to musl's vsscanf/strtol.
- amd64: point CC at musl-gcc so C deps compile against musl headers.
* iam/oidc: drop single-use web-identity tokens and the jti replay cache
Reverts the single-use semantics added in #11936 (and the #12017
refactor): AssumeRoleWithWebIdentity consumed each web-identity
token's jti into a bounded replay cache, so a second exchange of the
same token was rejected until exp + skew.
AWS STS accepts a web-identity token repeatedly until it expires,
and Kubernetes workload identity depends on that: the kubelet keeps
a projected ServiceAccount token until ~80% of its lifetime, so pod
restarts, co-located containers sharing the token file, repeated CLI
invocations, and pre-rotation SDK refreshes all re-present the same
jti and were refused on 4.50.
The replay machinery also carried a real maintenance cost — a
bounded concurrent cache, fingerprint-keyed entries for jti-less
tokens, adopt/merge handoffs so provider rebuilds could not reopen
consumed tokens, and provider-reuse bookkeeping in the refresh path
that existed only to preserve that state. Removing it restores AWS
semantics and deletes the whole apparatus.
Refs #12063. Alternative to the per-provider opt-out in #12064.
* iam: reuse unchanged OIDC providers on refresh
refreshOIDCProvidersFrom rebuilt every managed provider on any store
change, so an unrelated write discarded each provider's JWKS cache and
an unreachable issuer could then fail exchanges the cached keys would
have allowed. Keep a per-record provider cache keyed on the record
content fingerprint and only rebuild when the record changes.
* iam/oidc: require exp in ValidateToken
Dropping the single-use check also removed the implicit bound on
expiry-less tokens: a signed web-identity token without exp could
mint STS sessions indefinitely. Require the exp claim, matching AWS
AssumeRoleWithWebIdentity which rejects tokens without one.
* iam/oidc: bound idle connections on the provider transport
A provider replaced on refresh drops its http.Transport without a
close; give it an IdleConnTimeout so pooled connections and their
goroutines cannot linger.
* s3api: cover web-identity token reuse over HTTP
Sign one token and exchange that exact string twice, directly covering
the Kubernetes projected-token reuse this change restores.
* test(iam): refresh reuses an unchanged record's provider
Covers the provider-reuse cache: an unrelated record's refresh keeps a
provider's JWKS state, a changed record rebuilds it.
* s3: answer GET and HEAD of a delete marker as S3 does
A delete marker as the current version now returns 404 NoSuchKey with
x-amz-delete-marker: true and the marker's x-amz-version-id. Asking for the
marker by versionId returns 405 MethodNotAllowed with the same headers and
Last-Modified, instead of 404 NoSuchKey.
* s3: check the tag-conditioned read grant before answering a delete marker
The marker branch returned before the entry-based policy recheck the rest
of the read path performs, so an s3:ExistingObjectTag-conditioned Deny did
not apply to the delete-marker response and its version headers.
* s3: send Allow: DELETE with the delete-marker 405
RFC 9110 15.5.6 requires a 405 answer to list the methods the resource
supports; S3 sends allow: DELETE for a delete marker named by versionId.
* s3: keep delete-marker headers on conditional-read misses
If-Match or If-Unmodified-Since against a current delete marker short-
circuited to a bare NoSuchKey. Carry the resolved marker through the
conditional result so the 404 keeps x-amz-delete-marker and
x-amz-version-id, matching the unconditional read.
* s3: exercise delete-marker responses through GET and HEAD handlers
* s3: read marker version id from the entry before the caller's guess
A delete marker written outside the normal PUT path can miss the
version-id attribute while the v_<id> file name still names it; the
new x-amz-version-id header would then report "null". Reuse
versionIdFromEntry so both the handler and conditional-miss paths
emit the real id.
* s3: answer 405 for versionId=null on a suspended-versioning marker
The null marker lives in .versions while getSpecificObjectVersion only
checks the plain path, so GET/HEAD ?versionId=null answered
NoSuchVersion. On that miss, look up the null marker and answer it
like any other marker named by versionId.
* s3: propagate null-marker lookup failures instead of answering 404
A transient filer error reading .versions/v_null was swallowed as
"no marker", so GET/HEAD answered NoSuchVersion for a live read
failure. NotFound and absent entries still mean no marker; anything
else is InternalError now.
---------
Co-authored-by: Chris Lu <chris.lu@gmail.com>
Co-authored-by: chrislusf <chrislusf@users.noreply.github.com>
* filer: enable per-path commit records only when a peer has its own store
Since seaweedfs#11698 every metadata change writes a per-path commit
record into the filer store (an MpTs key, plus an MpDl key on deletes),
and the records stay after the path is deleted. They order the events a
filer replays from peer filers that keep their own store, since replay
only runs when peerSignature != f.Signature.
Filers that all share one store (one Redis, MySQL, TiKV, ...) never
replay each other's events, so the records are never read, yet each
change still costs one or two extra store writes and the key count grows
with every path ever changed — unbounded growth for in-memory stores.
This turns the records off by default and latches them on the first time
a peer reports a different store signature — no option needed. The latch
is persisted in the store (MpRep) so it survives restarts and converges
filers sharing a store even before they discover the peer themselves.
While off, a filer keeps an MpOff marker so enabling later bumps the
records' generation (MpGen, folded into each record key) past the
unrecorded window: a record older than a change made while off cannot
let an older replayed peer event win. Ordering for unrecorded changes
falls back to the entry's mtime, the same as entries written before the
records existed.
Supersedes the manual option approach in seaweedfs#12060.
* filer: keep delete tombstones while records off, harden enable path
Review on the first commit:
- A delete committed while the records were off left no entry and no
usable tombstone: after enabling, an older replayed create could
resurrect the path. Deletes and rename sources are now always recorded
under the generation-0 MpDl key — they are never fenced by a
generation bump, and they cannot go stale because every delete is
recorded. Creates/updates still write nothing while off.
- bumpPathCommitGen could regress MpGen under concurrent bumpers on a
shared store. It now loops: each pass writes only a generation higher
than any it has seen, so concurrent bumps converge upward.
- doSubscribeToOneFiler proceeded into replay when
enablePathCommitRecords failed, running the whole session without
ordering. It now aborts the subscribe so the reconnect retries with
the records on.
* filer: bind pending commit-record repairs to their generation
A failed MpTs write kept in metaTsPending could land after a shared-store
generation bump, writing a pre-bump timestamp as a current-generation
record and overriding newer unrecorded state. Each pending change record
now carries the generation it failed under: repairs write it only while
that generation is current, and ordering reads ignore it otherwise.
Pending delete records need no binding — MpDl keys are never fenced.
* filer: never move the local records' generation backward
The converge loop in bumpPathCommitGen seeded from zero and both it and
syncPathCommitGen stored unconditionally: a stale writer could leave the
stored generation below what this filer already adopted, and the next
sync or bump then walked the local value back — orphaning records it
wrote under the higher generation. The bump now seeds from the local
generation and every store only moves it forward.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* filer: name a records generation after the off marker it consumes
Two filers consuming the same off marker each counted one past the
stored generation, so the second could store N+2 after the first
stored N+1 and write records under a key the other could not read
until the next check. The bump now takes the marker's own timestamp —
filers racing on one marker land on one value — a marker older than
the stored generation is cleaned as a leftover, and a stored value
that already passed the bump is adopted instead of counted past. A
newly enabling filer whose marker was already consumed names its own
off window with the current time, so its unrecorded period is still
fenced.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* filer: tell a fresh off marker from one the generation already consumed
A marker whose timestamp reads at or below the stored generation was
dropped as leftover, but a new off period can also read that old — the
off filer's clock lagging, or a host clock moving back — and dropping
it kept records from before that period live, so stale replay could
win.
The stored generation now names the marker it was bumped for: a marker
the store still shows is consumed unless it is that one, however old
it reads, and only the consumed marker itself is cleared without a
bump. The generation trails in the value, so a whole-slice read still
yields it on stores written before the pair was stored. Enabling
fences the filer's own off window with the current time instead of a
marker a sharer may already have consumed.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* filer: cover racing off-marker consumers settling on one generation
* filer: adopt the stored generation, not the consumed marker it names
A filer syncing with no off marker present read the value's first field —
the consumed marker's timestamp — so a restart after a lagging-clock off
period under-counted the generation and kept records the bump dropped.
* filer: re-check the off marker before consuming it
A sync whose marker read ran behind a later off period's consume cycle
still bumped for its stale read: the stored generation already covered a
different marker, so the bump overwrote the consumed name, counted the
generation past records a newer consumption meant to keep, and cleared
whatever marker was live by then. Each pass now re-reads the marker and
only bumps or clears while it is still the one the pass saw.
* filer: keep a failed generation read from resetting the records' generation
The final generation read ignored errors and then stored whatever gen
held — possibly zero — so a transient store failure could hide records
the current generation wrote. Unexpected read errors now return before
the local generation changes.
Also cover a sync whose marker read runs behind another consumer's
delete: it must adopt the stored generation, not bump for the stale read.
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* filer: fail a bump that cannot verify the stored generation
A generation read that failed mid-bump was treated as absent: the
write could land below a generation a racing bumper had stored, or
re-store a marker that bumper had already consumed. Read failures
now abort the pass like every other store error on the path.
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* filer: cover generation reads that fail instead of returning a value
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* filer: carry the commit timestamp on the entry itself
The per-path MpTs record only exists because a commit timestamp had
nowhere else to live. Stamping commit_ts_ns on the entry before its
store write - and reusing that stamp as the emitted event's TsNs -
makes the entry its own commit record: replay and bootstrap traversal
order on it with zero extra writes for creates and updates.
Deletes still need a durable key since a removed entry cannot carry a
timestamp, so MpDl tombstones stay (one write per delete, growth
bounded by delete history). Entries predating the field fall back to
mtime, matching what peers and old records already assume.
This replaces the MpTs/MpOff/MpGen/MpRep machinery entirely: shared-
store filers write nothing extra, and dedicated-store replication
keeps last-writer-wins ordering plus tombstone coverage for deletes
and ancestor drops.
* filer: keep a directory update replay from tombstoning its subtree
A same-path update wrote a delete record before the delete-and-reinsert,
so a directory metadata change read as a subtree delete: a later peer
event for a child older than the update was then skipped even though
nobody deleted the children, leaving files missing on this filer. Only
file updates take the tombstone now — the re-inserted directory entry's
own commit stamp orders the path itself.
* filer: order unstamped entries by their recorded commit before mtime
Entries written before stamping existed have no CommitTsNs, so the
ordering check fell back to the client-settable mtime and dropped the
MpTs record those versions wrote: an entry committed at event time 300
with mtime 100 lost to a delayed peer event at 200 after upgrade. Read
the legacy record for unstamped entries in localCommitTsNs, CommitTsNs,
and the traverse stamp (which also stopped falling back to mtime,
regressing entry_ts_ns to 0).
* filer: release queued rename event stamps when the rename fails
Each moved entry carried an in-flight stamp its queued event would
adopt at emit time. When the move succeeded but a later child move or
the commit failed, the handler rolled back without emitting the queued
events or releasing their stamps, pinning the delivered and flushed
watermarks below the aborted rename forever. Release every queued
stamp on both failure exits in both rename handlers.
* filer: apply a directory update replay with an in-place upsert
Deleting the stored entry before inserting its replacement left a
window where the directory had no entry and no ordering record — a
tombstone cannot stand in for it without reading as a subtree delete
that suppresses the children's own records. A stale peer event accepted
in that gap could replay an older update or delete and sweep the
directory's children. Upsert the replacement instead: the entry keeps
its recorded commit order across the whole update, and a directory has
no chunks or hard links for a delete to release.
* filer: update an existing directory entry on replay instead of upserting
InsertEntry is not an upsert on SQL stores configured without upsert
support — PostgreSQL with enableUpsert=false issues a plain INSERT and
its duplicate-key error does not match the MySQL-only fallback, so a
replay updating a present directory failed on every retry. Pick
UpdateEntry when the entry exists and InsertEntry when it is absent;
UpdateEntry touches no row that is not there.
* filer: detect a shared-store delete racing a replayed directory update
UpdateEntry reports success even when the row was deleted between the
find and the update — the UPDATE changes zero rows and the replayed
change is dropped while the subscription offset advances. Re-read the
entry after the update and retry the write, re-checking the recorded
commit order each pass so a newer peer change wins instead of being
resurrected over.
* filer: declare commit_ts_ns in field-number order
Field 6 sits between extended and hard_link_id; the declaration now
reads in the same order. Wire format unchanged. Also syncs the Java
client's checked-in copy of filer.proto, which the pb Makefile copies
on gen.
* filer: option to skip delete tombstones entirely
filer.options.deleteTombstones=false stops writing MpDl records on
deletes - a single filer or shared-store cluster never replays peer
events, so the tombstones are dead weight. The tradeoff is explicit:
if a dedicated-store peer does replicate, a stale event can resurrect
a deleted path.
* filer: drop delete tombstones entirely
Deletes no longer write MpDl records, and the pending-repair machinery
they needed is gone with them: per-mutation metadata overhead is now
zero for every store topology. Ordering information lives only on the
entry itself via commit_ts_ns.
The accepted tradeoff: a dedicated-store peer replaying an event older
than a delete can resurrect the path, and a kept directory no longer
fences stale child creates. Recorded tombstones and MpTs records left
by older builds are still read, so upgrades keep the protections they
already have.
* filer: stop reading legacy per-path records too
Drops the MpTs/MpDl read path along with the writes: meta_path_ts.go
keeps only the entry-field readers (CommitTsNs, EntryCommitTsNs,
localCommitTsNs). Entries written before stamping existed fall back to
mtime, and tombstones or commit records older builds left behind no
longer fence replayed events.
* filer: trim commit timestamp comments
---------
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* mini: lift the post-mint allocation hold on first inventory report
DeferAllocation armed a fixed warmup window whenever the leader minted a
fresh topology id, so the first write on a brand-new weed mini blocked
for the whole window even though no prior cluster state could exist.
The window still has to cover a rebuilt master's veteran servers, which
report on their first post-reconnect heartbeat.
Mini has exactly one possible reporter — the embedded volume server,
whose first heartbeat carries the full inventory — so hold allocation
only until that report is applied. The bound raise and topology-id
freeze checks run while the heartbeat is processed, so a wiped master
state still raises the bound or trips the freeze before the hold lifts.
Standalone weed master and weed server keep the fixed window: volume
servers outside the process can still be on their way.
* master: apply queued topology-id reports before opening the allocation gate
SetTopologyInitPending(false) cleared the pending flag before replaying
topology ids stashed while the local id was still being minted. A
NextVolumeId landing in the drain window saw no gate and no freeze yet.
The post-mint allocation hold used to cover the gap by delaying the
winner past the freeze; once the hold can release on first inventory,
the gate must stay closed until the queued checks have run.
* mini: gate the first-report release on single-master mode
releaseAllocOnFirstReport assumes the embedded volume server is the only
possible reporter of prior cluster state. A mini started with
-master.peers joins other masters, whose own volume servers can still be
on their way when the first report lands, so keep the regular
quiet-pulse hold there.
* master: check reported topology id on empty full reports
A HasNoVolumes heartbeat carries the same LastSeenTopologyId a populated
list does, but the mismatch freeze only ran for non-empty reports. An
EC-only veteran could therefore release the post-mint allocation hold
through NoteInventoryApplied while its stale topology id never reached
CheckReportedTopologyId.
* master: drain the topology-id stash inside the init-gate lock
CheckReportedTopologyId only held reportedTopologyIdsLock while stashing,
so a report in flight during SetTopologyInitPending(false) could land its
freeze after topologyInitPending cleared, letting a NextVolumeId slip
through the gap. The whole check now runs under the lock, and the gate is
released inside the same critical section as the drain.
* mini, master: pin and enforce the single volume-server reporter
isSingleMasterMode only checks master peers; SendHeartbeat still accepted
any volume server, so a late foreign report could carry prior cluster
state after the first-report release dropped the hold. Mini now pins the
embedded volume server's address and the master refuses registration from
any other reporter while ReleaseAllocOnFirstReport is set.
* master: admit the volume-server reporter before applying heartbeat state
* test: cover SendHeartbeat admission ordering
A refused reporter must not freeze growth or raise the id bound, and the
pinned reporter's stale topology id still must.
* mini: re-derive the single-reporter release on every run
The package-level master options survive in-process reruns, so a
multi-master mini following a single-master one must not inherit the
first-report release or a stale reporter pin.
* master: admit addressless first heartbeats under the reporter pin
An unregistered heartbeat with no Ip skipped the single-reporter
admission check but still reached the seen-bound raise and the
topology-id freeze before the delta-heartbeat skip. Admission now runs
for every unregistered heartbeat; an empty address can never equal the
pin.
* util: add listenOnLocalhost parameter to NewIpAndLocalListeners
Nil keeps the current behavior of also binding localhost when the host
is a specific address; all callers pass nil for now.
* command: add -ip.bind.localhost flag to opt out of the extra localhost listener
Binding a command to a specific -ip.bind address also opened a second
listener on localhost for every port, with no way to turn it off. Each
command that calls NewIpAndLocalListeners (master, master.follower,
filer, s3, iam, volume, sftp, mq.broker, mq.agent) now accepts
-ip.bind.localhost=false to bind only the requested address. Embedded
services get prefixed flags (s3.ip.bind.localhost, sftp.ip.bind.localhost,
iam.ip.bind.localhost), and the command-level flag also disables them so
a single switch turns off every extra listener the command opens.
* mini: only require all-interface port availability while the loopback listener is on
With -ip.bind.localhost=false each service binds just -ip.bind, so a port
held on another address no longer collides; the all-interface precheck
would reject or relocate usable ports. WebDAV, the Admin UI, and the
Admin worker gRPC (which binds -admin.worker.ip, not -ip.bind) never open
a loopback companion listener, so their checks now use the address they
actually bind.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* mq.broker: bind the gRPC listener to -ip.bind instead of all interfaces
The listener used an empty host so -ip was ignored for binding and the
localhost companion was never created, leaving -ip.bind.localhost with
nothing to control. Give the broker the same -ip/-ip.bind pair the other
servers use; embedded mode binds to -ip.bind like its peers.
* server: add -sftp.ip.bind.localhost and -iam.ip.bind.localhost switches
weed filer already exposes the prefixed opt-outs for its embedded sftp
and iam; give weed server the same pair so each extra listener can be
disabled individually while -ip.bind.localhost=false still turns them
all off.
* mq.broker: keep the default all-interface bind unless localhost is off
An unset -ip.bind used to mean the standalone broker listened on every
interface; defaulting it to -ip silently cut off multi-homed clients. Only
fall back to -ip when -ip.bind.localhost=false, where a specific bind is
needed for the opt-out to take effect.
* mini: validate the Admin HTTP port on the address it will bind
The port check always used -ip.bind while startMiniAdminWithWorker falls
back to loopback when Admin has no credential, so a loopback-held port
passed validation and killed the admin at startup.
* mq.broker: keep the all-interface default bind when -ip.bind is unset
-ip only becomes the fallback once -ip.bind.localhost=false needs a
specific address for the listener.
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* mini: validate the Admin HTTP port on its effective bind IP
When the Admin listener falls back to loopback (no credential or mTLS
configured), the pre-flight port check now probes that same address
instead of -ip.bind.
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* util: keep NewIpAndLocalListeners source-compatible for 3-arg callers
* command: drop the -ip.bind.localhost flag; an explicit -ip.bind drops the extra listener
Instead of an opt-out flag, the loopback companion listener now simply
follows whether the bind address was chosen: it is created only while
-ip.bind is unset and the bind address falls back to -ip. Passing
-ip.bind=<addr> binds each port to exactly that address.
Each service options struct carries a listenOnLocalhost *bool that
embedders (weed server, weed mini, embedded s3/sftp/iam in weed filer)
set from the parent's -ip.bind state, captured before the bind flag is
repointed or filled in by the -ip fallback; standalone commands derive
it from their own flag before the fallback runs.
iam and mq.agent gain a real -ip.bind flag so they have a way to pin a
single address. mq.broker keeps its new -ip.bind flag but drops the
-ip.bind.localhost fallback hack: empty still listens on all interfaces.
* server: default embedded mq.agent brokers to the broker's bind address
With an explicit -ip.bind the embedded broker no longer listens on
localhost, so an unset -mq.agent.brokers defaulting to localhost:17777
could not reach it. Default to <serverBindIp>:<mq.broker.port> (the
address the embedded broker binds), falling back to localhost when the
bind address is a wildcard. An explicit -mq.agent.brokers still wins.
Also fix the NewIpAndLocalListeners skip message to say localhost.
* server: bind embedded mq.agent to serverBindIp
The embedded agent's ipBind was never set, so startQueueAgent fell back
to the advertised -ip address and an explicit -ip.bind did not pin the
agent listener.
---------
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat: send EC shards in parallel per destination disk
One stream per destination disk. max_shard_transfers unset means no global cap, 0 sends shards sequentially, and N is a global cap that still keeps one stream per disk. The value is persisted through TaskPolicy. The maintenance form is a checkbox plus Max transfers because unset and 0 differ. The key is intentionally omitted from GetConfigSpec.
* ec shard distribution: treat negative max_shard_transfers as sequential
The config layer (normalizedMaxShardTransfers) clamps negative values to 0,
which selects sequential distribution. shardTransferCap diverged: a negative
cap that reached ShardDistributionOptions directly would have selected
uncapped parallel mode, the most aggressive setting. Align the options layer
with the same normalization so invalid input degrades to the conservative
mode.
* ec: serialize shard transfers to a server when a target leaves disk choice to the receiver
disk_id 0 is auto-placement on the receiving volume server, so an
auto-placed stream can land on a disk another queued stream is pinned
to. Grouping by (destNode, disk_id) alone let both run at once; collapse
all of that server's transfers under the destNode key instead.
* worker: clamp max_shard_transfers to the int32 wire range
ReadIntConfig returned the 0 fallback for values above MaxInt32, turning
a large positive cap into sequential mode. Read int64 and clamp into
range instead.
* worker: drop unused shard-transfer form helper
The maintenance form renders and collects max_shard_transfers in
plugin.templ JavaScript; the Go copy had no callers.
* worker: bound max_shard_transfers conversion under an in-range guard
---------
Co-authored-by: Chepurnov Simon <195125934+CHEPURNOV-S-S@users.noreply.github.com>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
* filer, volume: ignore Range when If-Range no longer matches
A GET with Range and a stale If-Range validator got 206 with the requested
bytes of the current file. RFC 9110 13.1.5: send the whole content instead,
so a resumed download restarts rather than mixing two versions.
* volume(rs): drop Range when If-Range no longer matches
Mirror the Go fix for the Rust volume server: evaluate If-Range against the
response ETag (strong comparison) or Last-Modified once the needle's
validators are known, then drop the range on a stale validator so the whole
current representation is sent. A dropped range on the RangeFromSource
strategy falls back to a full stream of the data-file region; buffered and
chunk-manifest paths already take Option<&str>.
* volume(rs): fail If-Range when the header value is not readable
* volume(rs): accept all HTTP-date formats in If-Range like Go
* server: cover alternate HTTP-date formats in If-Range test
---------
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
* master: release the post-mint allocation hold after a quiet pulse
DeferAllocation armed a fixed warmup window whenever the leader minted a
fresh topology id, so the first write on a brand-new cluster blocked for
the whole window even though no prior state could exist. The window
still has to cover a rebuilt master's veteran servers, which report on
their first post-reconnect heartbeat.
Track the last heartbeat that could carry prior cluster state — a server
joining, held volumes or shards, a nonzero id bound, or a foreign
topology id — and lift the hold one pulse after it goes quiet, capped by
the original window.
* master: make the post-mint allocation hold configurable
master.volume_growth.allocation_deferral overrides the warmup window a
leader holds volume id allocation for after minting a fresh topology
id. Unset keeps the existing window; "0s" disables the hold for setups
that know the cluster is fresh (test harnesses, single-node labs); any
other duration replaces the cap while the quiet-pulse early release
still applies.
* master: re-check the allocation deadline while it is held
NextVolumeId slept for the whole computed delay, so a heartbeat landing
mid-sleep that pushed the quiet deadline out was ignored — the pending
allocation could still run ahead of a straggler's report. Wait in short
slices, re-evaluating the deadline and the growth freeze each round.
* master: count only ids above the issued bound as prior-state evidence
Heartbeat echoes of volumes and shards this master allocated during the
hold refreshed the quiet window, so each growth allocation re-extended
the deferral until the hard cap. Evidence now requires an id beyond the
bound this leader has issued, a higher reported bound, or a foreign
topology id; a joining server (dn == nil) still counts once.
* master: reject an unparseable allocation_deferral instead of disabling the hold
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* master: do not count a skipped heartbeat as a joining server
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test: assert the allocation hold extends past the original quiet deadline
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* master: release the post-mint hold after two quiet pulses
One pulse proved short for a veteran that reconnects on a staggered
restart: with only five seconds of quiet, allocation could resume and
reuse ids before a slower server landed its first heartbeat. Two pulses
still halves the fresh-cluster first-write stall versus the full warmup
cap, and known-fresh setups keep the "0s" override.
* master: pick once more when assign times out as growth concludes
The pick loop's final attempt can predate the growth it triggered
concluding: growth registers its volumes in the layout before clearing
the pending flag, so a timeout landing on that boundary used to shed a
hard 406 over volumes that had just arrived. Pick once more after the
loop in both the http and grpc assign paths before deciding how to
shed.
---------
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* s3tables: scope the resolved anonymous identity instead of defaulting to allow
Catalog authorization fell back to the open default whenever the request
resolved to the configured anonymous identity, ignoring its grant. The
fallback now applies only when no principal was resolved (zero-config),
so anonymous requests are checked against the identity's configured
actions like any other principal. The S3 Tables API and the Iceberg and
Lance catalogs share this handler, so all three are covered.
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test: authenticate the catalog clients in the s3tables suites
Anonymous calls into the Lance and Iceberg catalogs are now scoped by
the anonymous identity's grant, so the suites that used to rely on the
anonymous pass authenticate like their real deployments do: x-api-key
for the Lance probes and REST drivers, client-credentials OAuth for
pyiceberg.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* worker(lance): authenticate namespace calls with the storage credential
Anonymous catalog calls are now scoped by the anonymous grant, so a
worker that holds an S3 credential sends it as x-api-key on the
namespace requests instead of relying on the anonymous pass.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test: authenticate the pushdown catalog connection in catalog_lancedb
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* lance: follow catalog redirects only within the same host
reqwest forwards custom headers on a follow, so a redirect to another
host would carry the x-api-key credential to a server it was never
meant for. Keep same-host hops for proxies that normalize the path or
upgrade the scheme.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* lance: warn when the catalog api key travels in cleartext
The x-api-key pair is a real storage credential: over plain http to a
non-loopback host the network can read it. Warn at client build so the
operator sees it; https and loopback stay quiet, and the key still
sends since an http catalog is a valid deployment.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* lance: require the same origin on catalog redirects
A same-host hop can still downgrade https to http and put the api key on
the wire; matching scheme, host, and port keeps the credential on the
listener it was issued for.
* lance: bound catalog redirect hops so loops fail fast
The custom redirect policy drops reqwest's default 10-hop limit, so a
catalog that redirects back to itself would hold each call open until
the request timeout. Keep the bound alongside the origin check.
* lance: gofmt the redirect helpers
---------
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* admin: keep read-only sessions out of the filer /etc tree
The file browser read routes are open to any authenticated session, so a
read-only login could download or view arbitrary filer paths. The filer
stores the credential and cluster configuration under /etc (identity JSON
with plaintext S3 secret keys, SSE-S3 keys, remote storage config), which
let a view-only session recover live secrets that the user API redacts.
Gate the read paths on the session role: read-only sessions get 403 for
/api/files/download, /view, /properties, /metadata on /etc paths, the
browser page redirects away, and recursive metadata exports skip the /etc
subtree for read-only sessions.
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* admin: only reject real .. path components in file browser paths
path.Clean already resolves interior parent references, so any '..'
surviving in the cleaned path can only be a leading escape from a
relative input or part of a valid name. Matching on the substring
rejected legitimate entries like /buckets/reports/report..csv.
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* admin: cover the read-only /etc export exclusion with a test
Pull the excluded-prefix selection into exportExcludedPrefixes so the
role-dependent /etc skip is exercised directly — the cleanReadPath
tests alone would still pass if the exclusion were dropped.
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* s3api: stop echoing stored tags and internal attributes on GET/HEAD
setResponseHeaders copied every entry.Extended key into the response,
only filtering xattr- and x-seaweedfs-. Stored object tags
(X-Amz-Tagging-*) and the gateway's own attributes (Seaweed-X-Amz-Acl,
-Owner, -Version-Id, -Object-Lock-Mode, ...) therefore leaked to any
caller holding s3:GetObject, and to anonymous readers on public-read
objects, without s3:GetObjectTagging or READ_ACP.
Tag count, user metadata, storage class and the real S3 headers
(x-amz-version-id, x-amz-object-lock-*) are unaffected.
* s3api: match stored tag and internal attribute keys case-insensitively
GitHub is retiring ubuntu-22.04; move all 94 runs-on pins across 40 workflows to ubuntu-24.04. rust_binaries_release: the arm64 cross step was jammy-era — noble keeps apt sources in deb822 ubuntu.sources (the sources.list edit was a silent no-op that left amd64 repos open to arm64 and 404'd) and ports suites must say noble. Pin deb822 stanzas to amd64 and add noble ports entries.
2026-10-24 23:07:28 -07:00
805 changed files with 73442 additions and 10507 deletions
* master/filer/volume are stateful sets with anti-affinity on the hostname,
so your deployment will be spread/HA.
* leveldb2 is the default filer backend; a mysql-compatible database (memsql, ...) enables HA (multiple filer instances) and the backup/HA it can provide.
* with `filer.extraEnvironmentVars.WEED_MYSQL_ENABLED` set to `"true"`, mysql user/password are created in a k8s secret (default: `<release>-seaweedfs-db-secret`) and injected to the filer with ENV. On any other store neither the secret nor the `WEED_MYSQL_*` env, plain or secret-backed, is rendered.
* with `filer.extraEnvironmentVars.WEED_MYSQL_ENABLED` set to `"true"`, mysql user/password are created in a k8s secret (default: `<release>-seaweedfs-db-secret`) and injected to the filer with ENV. The secret is kept on `helm uninstall`, and a later install reuses the user and password in it, including ones you edited in. On any other store neither the secret nor the `WEED_MYSQL_*` env, plain or secret-backed, is rendered.
* cert config exists and can be enabled, but not been tested, requires cert-manager to be installed.
`namespaceOverride` deploys every namespaced object into an existing namespace other than the release namespace, for instance when the chart is a subchart. In-cluster addresses follow it. Set it before the first install: changing it later moves regular namespaced objects only — claims and their data do not move, hook resources such as the resize hook's ServiceAccount, Role, RoleBinding, and NetworkPolicy stay behind in the old namespace, and generated Secrets are not recreated in the new one. Migrate claims by hand and clean up the old hook resources. If the release uses generated S3 credentials, copy `<fullname>-s3-secret`, including its key fields, into the new namespace before the upgrade — otherwise the chart generates new keys and clients on the old ones fail authentication (explicit `s3.credentials` and `credentials.*.existingSecret` values do not use this generation path). With the default MySQL setup, also copy `<fullname>-db-secret` — a `pre-install` hook only, so it never reaches the new namespace — or pass `WEED_MYSQL_USERNAME` and `WEED_MYSQL_PASSWORD` through `filer.extraEnvironmentVars`. With the default `hostPath` storage the data is node-local too: stop the old workloads and plan the hostPath migration first, or the new pods can come up empty on other nodes or share the same directories concurrently.
## Gateway API
Every component with an `ingress` block (master, volume, filer, s3 and admin) can also be exposed through a [Gateway API](https://gateway-api.sigs.k8s.io/) `HTTPRoute`, for clusters that route through a Gateway instead of an Ingress controller. `<component>.httpRoute` sits next to `<component>.ingress` and is disabled by default. The chart does not create the Gateway, so point `parentRefs` at one that already exists, and the `gateway.networking.k8s.io/v1` CRDs must be installed.
@@ -606,6 +610,10 @@ Two things worth knowing before you turn this on:
The DNS selectors default to CoreDNS as kubeadm, kind and the managed offerings from AWS, Google and Azure install it. On OpenShift, override `egress.dnsNamespaceSelector` and `egress.dnsPodSelector` to match `openshift-dns`; see the comment in `values.yaml`.
## Common labels
`commonLabels` adds labels to the metadata of every object the chart creates. It may be a map or a template string rendering to one. The chart's own labels cannot be overridden, and pods (see `podLabels`) and `volumeClaimTemplates` are left unchanged.
## Pod and container security contexts
Pod and container security contexts are configurable independently for every built-in workload and remain empty by default for backwards compatibility. The examples in `values.yaml` show how to enable a `RuntimeDefault` seccomp profile, disable privilege escalation and privileged mode, drop all Linux capabilities, and use a read-only root filesystem.
Loaded 100 of 805 files, more files were not shown because too many files have changed in this diff.
Show more
Reference in new issue
Block a user
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.