With jwt.signing.read set, volume servers refuse unsigned reads, so
cp.from of a chunked file failed with "source returned status 401".
Mint a read token for the chunk on each source attempt, the same way the
filer's own reads do.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A follower took the scheme from [https.client], so with HTTPS clients
toward volume servers and plain-HTTP masters every forwarded request
(/dir/assign, /dir/status, the UI) failed with 502 "first record does not
look like a TLS handshake". Masters share one [https.master] setting, so
use HTTPS exactly when this master serves it.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
security.toml is mounted with subPath and read once at startup, so on an
upgrade that turns volume HTTPS on, the volume servers restarted with TLS
while filer, S3 and the other clients kept speaking plain HTTP. A
checksum/volume-https pod annotation, rendered only while the option is
on, rolls them all.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The copy built a bare client and plain http:// chunk URLs, so with
[https.client] on it could not read from or write to HTTPS volume servers
("source returned status 400"). Use the global client's transport and
redirect policy, and normalize the source and destination URLs to its
scheme.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Its volume server registers by pod IP, so HTTPS clients could not verify
the volume certificate.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Volume servers register as <pod>.<volume service>.<namespace>, and HTTPS
clients verify that name, which none of the certificate's wildcards cover.
With volume HTTPS on, the certificate also lists every volume group's
service.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
global.seaweedfs.volumeHttps.enabled sets [https.volume] to the volume
certificate and turns on [https.client] with the client certificate and the
chart CA, so the volume data port is no longer plaintext under
enableSecurity. The option fails without enableSecurity.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A scheduled run whose jobs no executor can take gives them up at once,
leaving them to the next detection. For a job type with a long detection
interval that is a whole interval later: a daily job whose only worker
was briefly full lost a day.
Bring the job type's next detection forward to within 10 minutes when
its jobs are given up for capacity, never pushing back an earlier one.
Claude-Session: https://claude.ai/code/session_011UpvafcfUuC2mLxNAXLQJi
* filer: renew AppendToEntry's cluster lock while the append runs
AppendToEntry took a 5 s short-lived lock and never renewed it, so an
append slower than that (slow store, slow volume, GC pause) lost the
lock while still running, and another filer could take it and append
to the same entry concurrently, dropping one append's chunks. Use the
auto-renewing lock the HTTP append path already uses.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* filer: refuse an AppendToEntry commit once its cluster lock was lost
A renewal can still fail, or the lock can be lost and granted again
under a new generation, while the append runs; another filer may have
appended in between. Check the lock and its generation right before
the entry is written, fail the append so the caller retries, and drop
the manifest chunks this attempt created.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* cluster: IsLocked reports a lock whose lease ran out as not held
isLocked only flipped when a renewal came back rejected, a second after
the reply, so a holder whose renewal stalled past the lease (a long GC
pause) still read as locked while the server could hand the key to
someone else. Track the lease from each grant's send time and require
it in IsLocked, which the filer append paths check before committing.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* filer: hold the slow-append test past the renewing lock's first lease
The renewing lock grants a 7 s lease, so a 6 s hold passed even with
renewal off; wait past LiveLockTTL so only a renewal keeps the lock.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* filer: say the AppendToEntry lock check is best effort
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* s3vectors: render filter string control characters as serde_json does
json_extract serializes the stored value with serde_json: \b, \f, \n,
\r, \t, \u00XX for the rest of U+0000-U+001F, and U+007F-U+009F raw.
The scalar comparison text used Rust debug escapes (\0, \u{1}, \u{7f}),
so $eq/$in on such strings missed scalar values and $ne/$nin kept them.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* test(s3vectors): filter on every control character, quote and backslash
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* vector: derive the indexed metadata key columns
An indexed metadata key is a hidden `_mk_<n>` column of List<Utf8>: the
strings an S3 `$eq` on a string matches, that is the value when it is a
string or the string elements of an array, and nothing for any other type
or a missing key. The field metadata names the key and its state
(`building` or `ready`).
The module derives the column from metadata given as JSON (the write
path) or as stored (jsonb or JSON text, for a backfill), and reads the
indexed keys from a schema. It lives in the worker crate so the engine and
the worker share it.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* vector: fill the indexed metadata key columns on PutVectors
PutVectors builds its batch against the table's current schema. Once a
table has indexed metadata key columns, the three-column batch no longer
matches that schema and the write fails. Fill each `_mk_*` column the
schema has from the row's metadata, whatever the key's state; a column
the engine does not know still fails the request.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* vector: answer string $eq and $in on a ready indexed key from its column
The gateway renders a string `$eq` or `$in` on a key as an OR of
`json_extract(metadata, P) = '"v"'` (or IN) and
`json_array_contains(metadata, P, '"v"')` over the same values. On the
parsed filter, a top-level conjunct of that shape on a key whose column is
`ready` becomes `array_has` / `array_has_any` on the column, which its
LABEL_LIST index answers. The two halves must name the same key and the
same strings, the scalar half in the text json_extract returns for them;
anything else, including `$ne`, `$nin`, `$exists`, numbers and booleans,
stays on the JSON path, and the other conjuncts still apply.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* vector: score few indexed matches exactly, per scope
With a ready indexed key in the filter, `ann` first counts the rows of its
scope that the key's conjuncts match, through the scalar index. When they
are at most the limit it scores them exactly over that scope, the way
the short-result fallback does; above it, the vector index searches with
the indexed prefilter, and the fallback still applies. The scope is the
one `exact_in_scope` scores, so per-scope exact answers merge into the
global exact top-k.
The limit is `Query::exact_match_limit` (None takes 256; 0 never counts),
set by the engine's --exact-match-limit (WEED_VECTOR_EXACT_MATCH_LIMIT).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* vector: count filtered queries by metadata key
For each index, the engine counts its QueryVectors calls by the metadata
keys of the parsed filter, with their total latency and the operand types
each key was compared with, over the last hour in 5-minute buckets. At most
512 indexes and 32 keys per index are kept (an idle key makes room;
otherwise the index is marked truncated), and the stats are not exported
as metrics labels.
The new FilterStats RPC returns an index's stats; the gateway's discovery
client forwards it like the other calls.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* vector: report the metadata columns capability in gRPC health
An engine whose writes fill the indexed metadata key columns reports the
gRPC health service `weed.vector.capability.metadata_columns` as serving,
so an operator action can refuse to index a key while an older engine
could still write the table.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* vector: expire filter stats' operand kinds and truncation with the window
The operand kinds a key was compared with, and the mark that a key went
uncounted, outlived the queries that set them. Keep the kinds per
5-minute bucket and the truncation as the bucket it last happened in, so
both leave the hour window with the counts.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* vector: fill indexed key columns a write left empty racing their add
A PutVectors that opened the table before an indexed key's column was
added, and commits after the add, is rebased by Lance onto the wider
schema: it commits, and the rows it wrote hold NULL in the new column, so
the column's index misses them (reproduced in the test: 100 of 150 rows).
After its commit, PutVectors now looks for indexed-key columns the batch
did not carry, reads those of its keys whose column is still NULL, derives
the values from their stored metadata and patches just those columns with
a partial-schema merge that updates only rows still NULL.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* vector: keep filter stats' truncation mark at the later bucket
record() reads the clock before it takes the lock, so a call that waited
on the lock across a bucket boundary could move the truncation mark (and
the index's last-used bucket) back to its earlier bucket, and the mark
then left the window one bucket early.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* vector: derive a raced write's indexed key columns from current metadata
A PutVectors that opened the table before an indexed key's column was
added is retried by Lance against the wider schema: rows it inserts hold
NULL there, and rows it updates keep the value derived from their old
metadata, which the NULL-only heal never looked at. The heal now derives
the columns for all of the write's keys.
Its merge also ran with Lance's retry, which joins the values derived
before the conflict onto the rows a later write of the same keys left.
It now fails on a conflict and is redone from the latest version.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* vector: check the raced write's keys keep the later write's rows
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* volume: apply a tailed size-0 record as a delete when the key exists
A delete and an empty write both reach the tail receiver as a size-0
.dat record. Writing every one as a live needle turned each delete
caught by volume.move's tail into a 200-empty object. Treat the record
as a delete when the key already has a record on the receiver, and as
an empty write otherwise.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* rust volume: apply a tailed size-0 record as a delete when the key exists
Mirrors the Go tail receiver: a size-0 record for a key that already
has a record here is a delete, otherwise an empty write.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* shell: volume.merge applies a size-0 record as a delete when the key exists
The tail stream carries deletes as size-0 records, and volume.merge
wrote each one to the merged copy as a live empty needle, so every
delete in the volume came back as a 200-empty object on all replicas.
A size-0 record for a key already written in the merged stream is now
a delete on the target; for a new key it stays an empty write.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* shell: volume.merge reads a size-0 record as a delete only after its own replica's record
Replicas append the same write at their own times, so a fresh empty write
can reach the merge twice when the copies are further apart than the
dedup window. Tracking keys across all replicas made the second copy a
delete. Track, per source replica, the keys it has returned (duplicates
the merge skips included): a size-0 record is a delete only when its own
replica wrote the key before, which is the only way a replica appends a
delete. A delete for a key the merged copy already deleted sends nothing.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* shell: volume.merge verification counts the deletes the merge applied
Applying deletes lets a correct merge hold fewer live needles than a
replica that missed a delete, and the live-count check then refused it
and kept the divergent replicas. Every key live on a replica is either
live or deleted in the merged copy, so compare that replica's live count
with the merged live count plus the keys the merge deleted. An empty
merged copy is still refused.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* volume: tail receiver looks a size-0 record's key up in the needle map
The receiver read the needle to decide whether a size-0 record deletes
an existing key. A read reports a needle past its TTL as not found, so
a delete for it was written as a live empty needle, which reads ignore
TTL for. Look the key up in the needle map instead (Volume.LookupNeedle).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* rust volume: tail receiver looks a size-0 record's key up in the needle map
Mirror Go: write_tailed_needle read the needle to decide whether a size-0
record deletes an existing key, and a read reports a needle past its TTL
as not found, so the delete became a live empty needle. Look the key up
in the needle map instead (Volume::lookup_needle).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* shell: volume.merge keeps a merged delete when a late empty copy follows
A replica that missed a delete can hold its copy of the empty write
stamped after the other replica's delete. As that replica's first record
for the key it was written, reviving the deleted key. A size-0 record now
never revives a key the merged copy already deleted.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* shell: volume.merge refuses more replicas than its key history has bits
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* wdclient: keep a volume's live replicas readable while another volume cannot be found
A filer's "volume not found" answer counted as a filer failure, so three
of them opened the circuit breaker on a healthy filer. With its only filer
in the cool-down, every lookup then failed without asking, as "all 1
filer(s) failed with non-retryable error: %!w(<nil>)". And a refresh whose
lookup failed left the cache without the volume's live replicas, since
InvalidateCache had already dropped them with the dead one.
The breaker now counts only failures to reach the filer, and asks the
filers anyway when all of them are in their cool-down. InvalidateCache keeps
what it removed; a lookup that errors puts it back unless the volume was
learned again, the epoch moved, or a reset would have expired it, and
leaves out locations fenced by a pushed removal.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* wdclient: set invalidated locations aside in the map instead of copying them out
The first commit kept what InvalidateCache removed in a side map and put it
back when the lookup failed. The copy left the map's lock, so a removal the
master pushed between the take and the put-back was undone: the removed
server came back into the cache.
The entry now stays in the map with a state. entryStale hides it from readers
and releases its server references, as a delete did. A pushed removal edits
it like any other entry. When the lookup after the invalidation returns an
error, the entry becomes entryRevived and is served again.
A stale or revived entry holds only what was known before the read failed,
so the next thing learned replaces it: a lookup answer drops it before the
answer is cached, including an answer with no locations and an answer to a
lookup that was in flight when another one failed, and a pushed add replaces
it instead of merging.
LookupVolumeIdsWithFallback returns no error when every volume it was asked
for has locations after the revive, so the filer's LookupVolume handler sends
its response instead of an error status.
The not-found breaker test now checks the failure count itself. Without
that, a regression of the classification passed, because a lone filer in its
cool-down is asked anyway.
On the first commit, TestPushedRemovalRacingFailedRefreshes failed in 8 of 8
runs under -race: four with "got [live:8080 old:8080], want [live:8080]",
four with "got [], want [live:8080]".
Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
* wdclient: retry a filer lookup pass that asked no filer
The pass decides once whether to skip unhealthy filers. A filer whose
breaker trips after that decision, as when a concurrent lookup records
its failure, is still skipped. With every filer skipped the pass ended
with no error and failed the lookup as non-retryable with
"%!w(<nil>)". Such a pass now counts as Unavailable, so the lookup
retries and the next pass asks the filers anyway.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* wdclient: believe a provider answer that reports errors for some volumes
The master provider answers a batch with a map of the volumes it knows
and an error naming the ones it does not. The failed-lookup branch took
that as no answer and revived what InvalidateCache had set aside, so a
volume the master reported as absent came back with its old locations,
and the batch could return them with a nil error. Only a nil map now
means no answer. Otherwise the answer replaces what was set aside, and
the provider's error is still returned.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* wdclient: keep set-aside locations learned after a lookup was asked
A successful lookup dropped every stale or revived entry of the volumes
it asked for, however old the answer was. A lookup asked before the
master confirmed new locations could, on finishing, delete those
locations after a read had set them aside and a failed refresh had
revived them, and its own answer was not cached because the epoch had
moved. Entries now record when their locations were learned, and an
answer drops only what was learned before it was asked.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* wdclient: keep an older answer from replacing locations set aside after it was asked
fdcde25 keeps dropSetAside from deleting set-aside locations that were
learned after a lookup was asked. The answer's insertion still replaced
them: fillLocations adds each location with vidMap.addLocation, and
addLocationToMap replaces a stale or revived entry whatever its learnedAt.
The epoch does not stop this when the newer locations came from another
lookup's answer or, on the master client, from a pushed add, since neither
moves it. The master client's provider also caches its answer itself,
before LookupVolumeIdsWithFallback reaches dropSetAside.
vidMap.addAnswer adds an answer's locations under one lock, and adds none
of them while vid has a stale or revived entry learned after the lookup was
asked. fillLocations uses it, and the master client's provider passes the
learned mark it takes next to its epoch. An answer with nothing newer in the
way still replaces what was set aside, with every location it reports.
On fdcde25, TestOlderAnswerDoesNotReplaceNewerRevivedLocations and the
"newer location revived during the lookup" case of
TestMasterAnswerAgainstSetAsideLocations fail with
"got [old:8080], want [new:8080]".
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* wdclient: move addAnswer next to dropSetAside
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
* admin/plugin: fail a restored job its worker no longer lists
An admin restart reloads the jobs that were running from the persisted
state, but the workers drop those jobs when the old admin stream closes.
No pending execution covers a restored job, so only the 24h stale sweep
ended it, and until then it blocked dedupe and scheduling of its job type.
Track the active jobs loaded at startup by worker and fail one after two
itemized heartbeats from its worker leave it out, as is already done for
a dispatch whose worker restarted. A listed job is left to adoption.
Claude-Session: https://claude.ai/code/session_011UpvafcfUuC2mLxNAXLQJi
* admin/plugin: keep tracking a listed restored job; skip queued ones
- A restored job its worker lists only resets its miss count, so one it
drops later is still failed.
- Track only jobs that were sent to a worker (request id set, not pending).
- Match restored ids against the trimmed heartbeat ids.
- Fail the restored attempt under the jobs lock that applies the expiry
(handleJobCompletedIf / expireActiveJobIf), so a newer attempt of the
same job is neither failed nor canceled.
Claude-Session: https://claude.ai/code/session_011UpvafcfUuC2mLxNAXLQJi
* admin/plugin: keep a newer attempt's detail from a restored-job expiry
- The conditional completion writes the job detail through
persistJobDetailSnapshotIf and skips it when the stored detail already
names another request; the check runs under jobDetailsMu with the write.
- mergeTerminalDetailIntoTracked ignores a terminal detail of another
request, so a restart does not end the loaded attempt with it.
- Tests wait for the restored-job expiries instead of sleeping.
Claude-Session: https://claude.ai/code/session_011UpvafcfUuC2mLxNAXLQJi
* admin/plugin: restore a finished retry from its detail
The job detail is saved as each attempt changes and the job list only
periodically, so a terminal detail naming another request is the newer
attempt. Apply it with its request id instead of ignoring it; the
expiry's own detail write already refuses to overwrite another request.
Claude-Session: https://claude.ai/code/session_011UpvafcfUuC2mLxNAXLQJi
* admin/plugin: restore whichever attempt is newer
The job list and a job's detail can name different attempts either way
round: the list is saved periodically, and a detail write can fail. When
their request ids differ, apply the terminal detail only if it is the
newer attempt (higher attempt number, else later update).
Claude-Session: https://claude.ai/code/session_011UpvafcfUuC2mLxNAXLQJi
* admin/plugin: fill a restored job's empty request id from its detail
A job the list saved while still queued has no request id; take the
finished detail's, as the merge did before the newer-attempt check.
Claude-Session: https://claude.ai/code/session_011UpvafcfUuC2mLxNAXLQJi
* admin/plugin: take the worker and attempt of a newer restored detail
When the restore takes a finished retry's detail over the job list's
attempt, the record kept the older attempt's worker and attempt number
next to the retry's request id and state.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* admin/plugin: fold the conditional completion into the existing functions
handleJobCompleted, expireActiveJob, trackExecutionCompletion and
persistJobDetailSnapshot take the accept check themselves instead of
gaining *If twins.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* admin/plugin: ignore a late earlier-attempt completion; replace an older detail
- A completion whose request id differs from the tracked attempt's is a
late report of an earlier attempt and no longer ends a retry.
- A conditional completion skips the detail write only when the stored
detail is a newer attempt (detailIsNewerAttempt), so an older finished
attempt's detail left behind by a failed write is replaced.
Claude-Session: https://claude.ai/code/session_011UpvafcfUuC2mLxNAXLQJi
* admin/plugin: take only the dispatched attempt's reports
A late progress report of a timed-out attempt replaced the retry's request
id, after which the retry's own completion was rejected as foreign. One
rule now gates both: a record the admin dispatched (attempt > 0) takes
progress and completion only for its own request (reportFromOtherAttempt).
When an expiry replaces an older attempt's detail it also takes the
closing attempt's worker, attempt number and times.
Claude-Session: https://claude.ai/code/session_011UpvafcfUuC2mLxNAXLQJi
* admin/plugin: restore a newer running attempt's identity from its detail
The job list can name attempt A while the detail shows retry B running
when the admin stopped. The restore now takes B's request id, worker and
attempt number from such a newer running detail, so B's own reports are
accepted. Test helper keeps a given attempt number.
Claude-Session: https://claude.ai/code/session_011UpvafcfUuC2mLxNAXLQJi
* admin/plugin: keep attemptEnded's doc comment on attemptEnded
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* admin/plugin: dispatch an attempt-less job as attempt 1
The late-report guard only covers records with a positive attempt, and
ExecuteJob passed a zero attempt through, so a direct caller's dispatch
could be closed by another request's completion. The HTTP API already
raised zero to one; the dispatch path now does the same for every caller.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* filer: keep a hard-linked file's chunks when one of its names is replaced
Replacing one name of a hard-linked file deleted the file's chunks while
its other names still pointed at them. Reads through those names then
return EIO once the volumes are vacuumed; until then they only work
through readDeleted.
Two everyday ways to get there through a mount:
- conda links an environment's files to its package cache with hard
links. Python saves a __pycache__/*.pyc by writing a temporary file and
renaming it over the old name, so the cache's copy loses its data.
- A training script saves model.pt, hard-links it into an epoch directory
and saves the next epoch by renaming a new file over model.pt. Each
archived epoch loses its data.
Three write paths deleted the chunks of the entry they replaced without
asking whether that entry was one of several hard links:
- a rename over a file: moveSelfEntry drops the target's link, then
queues all of the target's chunks for deletion;
- CreateEntry over an existing path (an S3 PUT, for one): the update
drops the old entry's link, then deleteChunksIfNotNew deletes its chunks,
and the CreateEntry handler deletes chunks its own list covers;
- UpdateEntry with an entry that no longer carries the old one's link.
A plain delete already keeps the chunks while HardLinkCounter > 1, and a
folder delete asks hardLinkIsOrphaned. The three paths now go through
Filer.KeepHardLinkedChunks, which asks after the write, once the replaced
name's link is gone: the chunks stay unless the file's shared record went
with it, that is, unless the replaced name was the last link. Writing the
file through one of its names (an entry with the same hard link id) still
deletes the chunks the write replaced, since every name reads the new
content.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* filer: keep hard-linked chunks from the shared record, not the replaced row
A lookup that cannot read the hard link record returns the name's own row,
which can lag the file. Read the record when filtering the garbage, and keep
everything when it cannot be read.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* filer: test the manifest reads behind KeepHardLinkedChunks
The manifest resolution moves into withoutChunksOf, which takes the lookup
function, so tests can serve manifests. Chunks behind nested manifests and
the manifests themselves stay, unrelated garbage is still deleted, and an
unreadable manifest returns an error, on which KeepHardLinkedChunks deletes
nothing. Behavior is unchanged.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
* volume: count needles vacuum drops as unreadable and log each at error level
Vacuum drops a live needle whose record fails to read back intact (CRC
mismatch, malformed fields, short or dangling record). The key leaves the
compacted index with no tombstone, so repair tools later treat it as a
vacuumed delete. Log every dropped key at error level and count the drops
in SeaweedFS_volumeServer_vacuuming_dropped_needles per collection, once
the compacted index is written.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* grafana: add a panel for needles dropped by vacuum
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* seaweed-volume: count needles vacuum drops as unreadable and log each at error level
Mirror the Go volume server: log every dropped key at error level and add
SeaweedFS_volumeServer_vacuuming_dropped_needles per collection once the
compacted index is written.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* volume: count dropped needles only after the compacted dat is flushed
The final .cpd fsync runs in the deferred close and loadTmpNeedleMap can
still fail after the index is saved; a copy failing there is retried and
would count the same needles again.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* filer: lock the path for plain HTTP overwrites
CreateEntry looks up the entry it replaces and queues that entry's chunks
for deletion. Two plain PUT/POST writes of one path could both read the
same old entry, so the chunks of whichever write lost were never queued.
Take the exclusive path lock around CreateEntry, as the gRPC CreateEntry
already does.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* filer: test concurrent HTTP overwrites queue every replaced chunk
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* filer: resolve deleted manifests through the master on a volume cache miss
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* filer: keep a manifest needle until its sub-chunks are queued for deletion
When a manifest could not be read during chunk deletion, the manifest
needle was deleted anyway and the data chunks it listed were orphaned.
Retry the resolve with the deletion retry backoff and delete the
manifest alone only after MaxRetryAttempts.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* filer: keep a durable delete that joins a pending manifest retry
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* filer: bound each manifest retry read with a deadline
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* filer: requeue resolved manifest ids on an fsync upgrade instead of rereading
A durable delete that joins a manifest retry in flight used to resolve
the manifest a second time, but the first pass had already queued the
manifest needle, so the reread could fail and the sub-chunks never got
the durable mark. Resolve nested manifests up front and queue the same
ids again with fsync.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* filer: bound each manifest retry pass and stop it on shutdown
A pass takes ready manifests one at a time and stops after
manifestRetryTimeout, leaving the rest queued for the next tick, so a run
of stalled reads cannot keep the shared retry loop from ordinary deletion
retries or from noticing shutdown.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* http: report the status of a failed delete that has no body
A replica that answers a DELETE with a bare 500 left the forwarding
server reporting "[<replica>]: " with an empty reason.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* volume: forward a delete retry that finds nothing here to the replicas
A delete is applied locally before it is sent to the other replicas, so
when the fan-out fails the client is told to retry while this replica
already dropped the needle. The retry then read nothing here and answered
404 without asking the replicas, so the replica that missed the delete
kept the needle for good.
On a replicated volume, a plain delete that finds the needle missing or
deleted now cancels a write still queued for it, as a replica delete
does, and sends the delete on to the other replicas. A failed fan-out is
reported as before, so the client retries until every replica agrees.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* volume(rs): take a replica's 404 to a forwarded delete as done
A replica that no longer holds the needle answers a forwarded delete with
404. The Go server folds that into success, but the Rust server reported
it as a replication failure, so a delete that only had to catch up one
replica failed on the replica that was already in step.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* volume(rs): forward a delete retry that finds nothing here to the replicas
Mirrors the Go server: on a replicated volume, a plain delete that finds
the needle missing or deleted cancels a write still queued for it and
sends the delete on to the other replicas, so a retry after a failed
fan-out reaches the replica that kept the needle.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* volume: answer a delete whose needle read fails with 500, not 404
A replica's 404 to a forwarded delete is taken as the needle being gone
there. DeleteHandler answered 404 for any failed pre-read, so a replica
whose disk could not read the needle let the primary acknowledge a delete
that replica never applied. Only a missing or deleted needle (or a volume
this server does not hold) answers 404 now; other read errors are a 500,
so the client retries.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* volume(rs): answer a delete whose needle read fails with 500, not 404
Mirrors the Go server: a replica's 404 to a forwarded delete is taken as
the needle being gone there, so only a missing or deleted needle answers
404; other read errors are a 500 and the client retries.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* volume(rs): corrupt the needle with portable file writes in the delete read test
FileExt::write_all_at is Unix-only, so the test did not build on Windows.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* volume: serve decoded bytes for a range over a compressed needle
A Range with Accept-Encoding: gzip got Content-Encoding: gzip and a slice
of the compressed bytes, which a decoding client cannot read. The range
is over the stored file's decoded bytes, so pass gzip through only for a
whole read.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* seaweed-volume: serve decoded bytes for a range over a compressed needle
Mirrors the Go volume server: a Range with Accept-Encoding: gzip got
Content-Encoding: gzip and a slice of the compressed bytes. Pass gzip
through only for a whole read.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* seaweed-volume: HEAD of a compressed needle reports the served length
HEAD answered a compressed needle from its meta: the stored gzip length
with no Content-Encoding, whatever the request accepted. Go reads the
payload and reports what the GET would send: the decoded length, or the
gzip length with Content-Encoding: gzip for a whole read that accepts
gzip. Read a compressed needle in full on HEAD too.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* test/volume_server: range and HEAD over a compressed needle
Both volume servers serve a gzip-accepted range decoded, and HEAD reports
the length the matching GET would send.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* seaweed-volume: decode every gzip member of a compressed needle
Go's gzip reader reads concatenated members; GzDecoder stopped after the
first, so a range over such a needle saw a shorter body and length.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* vector: answer a short filtered ANN query by an exact scan
A prefiltered query through the IVF index sees only the partitions it
probes. When those hold fewer than k eligible rows the answer is short
(400 matches, topK 500, 63 partitions: about 208 returned), and when at
most k rows match Lance appends the ones its probes missed with an
infinite distance. Either case is now answered again by an exact scan of
the same scope: the listed fragments plus the live coverage of the listed
index segments, or the whole table.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* vector: count an explicit probe count as indexed only with an index
With nprobes set on a table that has no vector index, the scan was marked
indexed, so a filtered query matching fewer than k rows ran the same exact
scan a second time.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* filer: retry TiKV store writes on a write conflict
TiKV transactions are optimistic. A commit fails with a write conflict
when another transaction committed one of its keys after it began, even
for a blind Set or Delete. Outside a filer transaction the TiKV store
runs each save and delete in a transaction of its own and returned that
error to the caller, so of two requests writing the same entry at the
same moment one failed, and a mount reported EIO.
A write the store commits itself (InsertEntry, UpdateEntry, DeleteEntry,
one batch of DeleteFolderChildren, KvPut, KvDelete) now runs again in a
new transaction after a write conflict, up to 5 times, waiting 1-2ms,
2-4ms, ... 16-32ms in between. The failed transaction stored nothing,
and the retry lands after the other writer, as if the two had not
overlapped. The retries stop when the context is done.
A write inside a transaction from BeginTransaction is not retried: the
conflict is reported when the caller commits, as before.
deleteBatch now goes through RunInTxn like the other writes.
Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
* filer: stop TiKV write retries once the context is done
When the context was cancelled while the retry wait was running, the
timer case could still win the select and start one more transaction.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* filer: keep a TiKV entry that is no longer expired when listing deletes it
The listing decides an entry has expired from its own read, and the
delete ran later as a blind delete. Once the delete retries after a
write conflict, a live replacement committed in between was removed.
Read the entry again in the deleting transaction and delete it only if
it is still expired.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: MapleWhiteZZZ <18653045+MapleWhiteZZZ@users.noreply.github.com>
Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
* storage: reject a streamed read whose record id or size differs
The header probe before a streamed read returned not-found only when both
the record's id and its size differed from the index entry, so a record of
another needle with the same size passed and its data was streamed under
the requested key until the final checksum check.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* storage: refuse a needle blob whose header names another needle
WriteNeedleBlob compared only the blob header's size with the size it was
asked to index. A blob read from an offset that has since moved to another
record of the same size was appended and indexed under the wrong key.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* volume(rs): refuse a needle blob whose header names another needle
Mirror of the Go check: write_needle_blob_and_index compared only the blob
header's size, so a blob of another needle with the same size was appended
and indexed under the requested key.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* volume(rs): report not-found when a streamed read's record is another needle
The meta read ahead of a streamed reply took the id from the record header
without comparing it with the requested key. Return not-found when the
index entry resolves to another needle, as the Go header probe does.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* storage: check the requested id after a meta-only read
ReadNeedleMeta overwrites n.Id with the record's header id, so a meta-only
read through an index entry that resolves to another needle returned that
needle's metadata, and the streamed-read probe that follows compared the
record with itself. Return not-found when the header names another needle,
as the Rust meta read does.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* test: copy a needle blob to another volume, not under another key
WriteNeedleBlob now refuses a blob whose header names another needle,
so the round trip writes the blob under its own id into a second volume
and checks that storing it under another key is refused.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* storage: compare the streamed header with a deleted entry's record size
A deleted index entry stores the negated size, so a read-deleted stream
compared the positive header size with a negative one and missed.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* storage: copy an empty needle written during compaction in makeupDiff
A size-0 index row is a live empty needle, not a delete. makeupDiff
grouped it with tombstones and appended a made-up delete marker, so
the compacted .dat lost the record's cookie and held a delete.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* volume(rs): copy an empty needle written during compaction in makeup_diff
A size-0 index row is a live empty needle. makeup_diff wrote a made-up
delete marker and a tombstone row for it, so the needle read as not
found after the commit. Copy the record like any other live row.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* storage: fail makeupDiff when the replayed needle write fails
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* volume(rs): check the compacted empty needle's id and cookie in the test
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* admin: show a pruned job's detail from its run record
Tracking keeps the newest 1000 jobs across all job types and deletes
the details of older finished ones, while each job type's run history
keeps its last 10 successful and 10 failed runs. A job listed in the
run history could therefore open as "job detail not found". Fall back
to the run record when the job is no longer tracked.
Claude-Session: https://claude.ai/code/session_011UpvafcfUuC2mLxNAXLQJi
* admin: show 100% progress for a pruned successful run
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* admin: use the newest run record for a job across both outcomes
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* admin: keep the run record a pruned job's detail was built from
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* admin: keep looking for a run record past an unreadable history
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* admin: pick a pruned job's newest run record across job types
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* storage: red test for CommitCompact after CompactByVolumeData with a concurrent write or delete
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* storage: build the temporary needle map in CompactByVolumeData
Fixes#12783
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* storage: require a successful commit in the CompactByVolumeData commit test
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
BinarySearchByAppendAtNs, its left/right probes and ResumeByLastEntry read
every probed record through readNeedleAt, which parsed the whole body and
verified the data checksum, although they use only the id, cookie, size and
append time. One record with damaged data on the search path, even a needle
deleted long ago, failed the search and with it the tail sender, a
timestamp-based incremental copy and the volume.move catch-up.
readNeedleAt now reads the header and the trailer and skips the data. The
data checksum is still verified where the data is read, including the tail
receiver for every record it gets. The Rust volume server already parses
only the metadata on this path and does not verify the data checksum.
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* storage: red test for the append-time search racing a vacuum commit
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* storage: hold the data-file read lock across the append-time search
Fixes#12785
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* storage: Version() writes the superblock version only when it changes
Version() stored the .vif version into SuperBlock.Version on every call, so
each call was a write that raced with readers of the superblock that do not
take superBlockAccessLock, such as the append-time search and the
compaction setup.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* storage: harden the append-time search race test; add a lock-hold budget test
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* storage: Linux-only lock budget; stop before reading search failures
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
* security: add a constant-time HTTP Basic Auth check
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* filer: require HTTP Basic Auth on the HTTP ports when configured
With filer.basic_auth.password set in security.toml (or
WEED_FILER_BASIC_AUTH_PASSWORD), a request on the filer HTTP ports, TUS
included, must carry those credentials or a valid filer JWT, and an
access level with no JWT signing key is no longer open. Valid Basic
credentials grant read and write. 401 responses then carry a Basic
challenge so browsers prompt. The credentials reload on SIGHUP with the
signing keys.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* scaffold: document filer.basic_auth in security.toml
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* filer: enable reloaded Basic Auth before swapping the JWT keys
Reload cleared the old signing keys before storing the new Basic
credentials, so a switch from JWT to Basic Auth briefly left both off
and admitted anonymous requests. Store new credentials first and clear
old ones last.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* filer: refuse Basic credentials on cross-origin browser requests
Browsers resend cached Basic credentials on requests other websites
trigger, such as a form POST with mv.from, and the default
cors.allowed_origins "*" lets them through. Accept Basic credentials
only on requests without an Origin, from the filer's own host, or from
an origin explicitly listed in cors.allowed_origins. A JWT is still
accepted from any origin.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* filer: recheck Basic Auth after reading an empty signing key
A request that loaded Basic Auth as off, then read the signing key after
a JWT-to-Basic Reload had cleared it, was let through anonymously. Open
access now also needs Basic Auth to be off after the key read; with
Reload enabling the new credential before disabling the old one, the
two loads cannot both miss a credential that was on throughout.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* security: compare Basic Auth credentials as MACs under a per-process key
A plain SHA-256 of the password is what password-hash scanners flag;
keying the digests with a random per-process HMAC key keeps the
length-independent constant-time compare without holding one.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* util: read a configuration file without merging into the shared viper
LoadConfiguration merges each read into the shared viper, so a key
removed from the file keeps its old value after a reload.
ReadConfiguration reads the file into a fresh viper with the same search
path and WEED_ environment overrides.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* filer: turn Basic Auth off when security.toml no longer sets it
The shared viper merges each SIGHUP reload into what it held, so
removing [filer.basic_auth] kept the old password valid. Read the
credentials from security.toml alone at startup and on reload; an
unreadable file keeps the current setting.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* filer: keep Basic Auth when security.toml cannot be read
A malformed file at startup dropped a WEED_FILER_BASIC_AUTH_PASSWORD
from the environment, and viper reports an unsearchable config
directory as a missing file, which turned Basic Auth off on reload.
ReadConfiguration now always returns the environment overrides and
reports whether the file was read; a reload that could not read it
keeps the current credentials.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* filer: keep reloading Basic Auth from the security.toml first read
A search directory that turned unsearchable made viper fall through to a
lower-priority security.toml, and a reload from a file without
[filer.basic_auth] turned Basic Auth off. ReadConfiguration now takes the
file to read, and the filer reads the file it first found on every reload.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* wdclient: sort newly discovered filers and log every current-filer switch
Discovery appended new filers in map iteration order, so after the current
filer left, the fallback at index 0 was a random filer. Append them sorted.
Route every change of the current filer through one helper that logs it at
V(0) with its reason (failover, return_home, discovery), so an operator can
see which filer a gateway uses without new metrics.
* wdclient: prefer a home filer and return to it after failover with backoff
A FilerClient with FilerClientOption.HomeFiler set goes back to that filer
once its circuit breaker allows a re-check, instead of staying on whichever
peer it failed over to. Leaving home again soon after a return doubles the
wait before the next return, up to 5 minutes; a return that holds for 10
minutes resets it. Volume lookups and discovery follow the same preference.
* s3: embedded gateway uses its local filer as home filer
weed filer -s3, weed server -s3 and weed mini now build the gateway's filer
address in the form the filer registers in the master (ip:port.grpcPort),
with the gRPC port default applied, and pass it to the gateway as its home
filer, so the gateway returns to its local filer after any failover.
Standalone weed s3 has no home filer and behaves as before.
Refs #12768, follow-up to #12771
* wdclient: switch the current filer only from the filer a request started on
A volume lookup that started on a peer and finished after the client had
returned home moved it back to the peer and counted that as leaving home,
doubling the return backoff although home never failed. Concurrent
failovers could also each count as a separate departure.
Failovers now name the filer they started on and take effect only while it
is still current, via compare-and-swap, so only the winning switch logs and
feeds the backoff. S3's failover walk uses the same rule.
* wdclient: switch the current filer and record the return backoff in one step
The index moved by compare-and-swap before the switch was recorded, and
both ran under the shared read lock. A request reading the current filer
in that gap saw the peer with the old eligibility time and, with home's
breaker still closed after one failure, switched straight back home.
The eligibility check, the move and the bookkeeping now all run under
homeMu.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* wdclient: discovery falls back past a home that is still backing off
When discovery removed the current filer and home was not yet eligible,
the client took index 0, which is home whenever it is listed first. With
home's breaker still closed after a single failure, requests went back to
the filer they had just left. Take the next filer instead.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* wdclient: start return timing on every switch back to the home filer
Only a return_home switch stamped lastReturnHome. Discovery or a peer's
failover could also put the client back on home, and a failure soon
after such a return kept the backoff at its base value instead of
doubling it.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Konstantin Lebedev <whitefox@mayflower.work>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* filer: recover leveldb without rebuilding broken tables
leveldb.Recover rebuilds a table with damaged blocks using the user
comparer and filter on internal keys. The shortened index keys it writes
are not valid internal keys, so a Get on the rebuilt table panics, and
the bloom filter it writes misses every lookup. Recover with a comparer
that keeps full index keys (same name, so the manifest still matches)
and no filter.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* filer: recover a leveldb store when a compaction finds a corrupted table
When a compaction reads a damaged table block, as a power loss can
leave one, goleveldb latches the corruption error and returns it for
every later write until the database is reopened. A plain reopen loads
the same table and the next compaction fails again, so the store kept
rejecting inserts even across restarts.
Treat a corruption error from a write like a storage error, and reopen
through leveldb.Recover, which rebuilds the table without the damaged
blocks, then retry the write.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* filer: rebuild damaged leveldb tables before Recover
leveldb.Recover's own table rebuild orders internal keys with the user
comparer, so a damaged table holding several versions of one key fails
with "keys are not in increasing order" and the recovery gives up.
Overriding the comparer cannot fix that, because the same comparer
also serves the recovered database's user keys.
Rebuild every table that has corrupted blocks first, using an internal
key comparer and full index keys, then let Recover rebuild the manifest
from tables it no longer needs to rewrite. Seed the test tables from a
journal flush so they keep several versions of a key, and check keys
the write test does not rewrite.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* filer: leave unreadable leveldb tables for Recover to drop
table.NewReader reports a truncated table or a damaged footer through
the reader, not its error, so rebuildDamagedTables never stopped on one,
but it rewrote it as an empty table first. Skip tables with nothing
readable and let Recover drop them, and test an unreadable table next
to a healthy one.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* filer: let StrictRecovery drop damaged leveldb tables
With StrictRecovery set, leveldb.Recover drops a table that has corrupted
blocks. Salvaging the readable blocks first would hide the damage from it,
so skip the rebuild under that option.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* filer: hold the leveldb storage lock while rebuilding tables
The table rebuild ran after the failed open released the storage lock
and before Recover took it, so it could rewrite the tables of a database
another caller had open on the same storage. Take the lock for the
rebuild; Recover takes it again afterwards.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* master: reject an assign on a follower as Unavailable
Assign on a master that is not the raft leader returned
raft.NotLeaderError as a plain error, which reaches gRPC clients as
codes.Unknown. operation.Assign retries only Unavailable,
ResourceExhausted, Canceled and DeadlineExceeded, so every assign made
during a leader election failed at once instead of waiting for the new
leader.
Return it as codes.Unavailable, like the warming-up status in the same
file, so clients retry within their 30 s assign budget. The message is
unchanged, so callers that match "Not current leader" still work.
StreamAssign now answers such a request in-band with the error message
instead of ending the stream, as it already does for warm-up.
TestAssignOnFollowerIsUnavailable covers it.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* pb: keep the cached channel when the peer answers Unavailable with details
gRPC's own Unavailable (dial failure, closing transport) never carries
details, so one that does came from the peer over a working channel.
Closing the shared ClientConn for it cancels every other in-flight RPC
on it with "the client connection is closing".
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* master: tag the follower's assign rejection with a NOT_LEADER ErrorInfo
The detail marks it as the master's answer, so clients retry it
without tearing down their cached channel to that master.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
* test: let the merge-read vacuum outlast the master's write drain
Before compacting a volume the master waits up to 30 s for writes it
assigned to that volume to drain. The test cut each /vol/vacuum call at
30 s and gave the vacuum 60 s, so one or two slow drains failed it.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* test: retry the merge-read vacuum until the master runs one
A 409 means another vacuum holds the master's lock; check the chunks
only after a vacuum this test asked for has finished.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* util/http: probe an unanswering server in the background, not on a read
Once the retry interval passed, the first read put the dead server first
and waited out the full dial timeout before trying a live replica. Keep it
last and dial it from a background probe with a short timeout instead.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* util/http: try every server of a machine that does not answer last
Unreachability was kept per host:port, so a powered-off node running one
volume server per disk cost one stalled read per server before each was
known dead. A dial that times out or finds no route now marks the whole
machine; a refused connection still marks only that port and shows the
machine is up.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* util/http: probe an unanswering server with an HTTP request
A bare TCP connect cleared servers that accept connections but never send
response headers, and could not dial an address without an explicit port.
Probe with a HEAD request through the global client instead, and have the
tests wait for the probe to settle.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* util/http: keep a timed-out port last after a sibling port answers
A dial timeout stored only the machine verdict, so a success on another
port of that machine cleared it and the port that timed out went back to
the front. Record the port as well as the machine.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* util/http: classify refused and unroutable dials with Winsock errnos on Windows
Windows reports these as WSAECONNREFUSED, WSAEHOSTUNREACH and friends,
which errors.Is does not match against the syscall constants. Test the
refusal with a real dial to a closed port.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* util/http: wait for background probes through a WaitGroup in tests
A refused probe can finish before the test reads the claimed record, so
inferring completion from the record's timestamp could wait out the
deadline on a probe that already settled.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* util/http: key machine verdicts apart from hosts without a port
A host without an explicit port is also its machine name, so a sibling
port answering cleared the timed-out host along with the machine verdict.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* util/http: group a bracketed port-less IPv6 host with its machine
net.SplitHostPort rejects "[2001:db8::9]", so the machine key kept the
brackets and missed the verdict stored for "[2001:db8::9]:9501".
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* filer: metrics for metadata subscription delivery (lag, holds, ring fall-offs, claim and watermark ages)
* filer: measure subscribe delivery lag after the real send using the mutation stamp, and record a watermark hold once per hold
* filer: skip empty progress markers when sampling subscribe delivery lag
* filer: assert subscribe delivery lag is sampled only after the stream send succeeds
* grafana: panels for the metadata subscription delivery metrics
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* filer: record a watermark hold when the read passes the held entry, not when the next hold starts
* filer: use a monotonic clock for the subscribe hold duration
---------
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* filer: keep an explicit gRPC port in the embedded gateways' filer address
runFiler handed the embedded S3/WebDAV/IAM/SFTP gateways
NewServerAddress(...).String(), and String() drops the .grpcPort part.
With a non-default -port.grpc the gateways then dialed port+10000 and
did not recognize their own filer in the master's list.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* wdclient: match discovered filers by gRPC address
A filer listed as host:port and the same filer discovered as
host:port.grpcPort were compared as raw strings, so a refresh pruned the
configured filer, added every discovered filer as new in map order, and
reset the current filer to whichever came first; a pushed join added it
a second time. Key filers by their gRPC address: a survivor keeps its
health and current position and takes the discovered address form, and
pushed joins and leaves match the same way.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* security: scoped read-only token for filer metadata subscriptions
A token signed with jwt.filer_signing.subscribe.key names the subscribe
methods and the path prefixes its holder may see. mTLS peers and admin
tokens stay unrestricted. Internal subscribers attach an admin token
when both keys are configured.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* filer: enforce subscription token scope on metadata streams
With jwt.filer_signing.subscribe.key set, SubscribeMetadata and
SubscribeLocalMetadata authorize the caller first. A scoped caller gets
a stream that sends only events whose paths are all under its prefixes,
filters batches, and never carries log file refs.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* filer: send an admin token on internal metadata subscriptions
Peer filers, metadata followers and the lifecycle reader attach an
admin token when the subscribe key is configured, so they keep the full
stream without mTLS.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* shell: add fs.meta.token to mint metadata subscription tokens
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* filer: end scoped metadata subscriptions at token expiry
The scope now carries the token's exp. The scoped stream's context ends
at that time and the call returns Unauthenticated, so an open stream no
longer outlives its token.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* filer.sync: sign metadata subscriptions with the source cluster's key
MetadataFollowOption takes a FilerJwtProvider; filer.sync passes the
one loaded from -a.security or -b.security, so clusters with different
admin keys both accept the subscription.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* shell: report fs.meta.token flag errors
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* filer: give scoped metadata subscribers their own client id
A scoped caller could reuse another subscriber's ClientId with a higher
epoch and replace that registration, disconnecting it. Scoped streams now
register under a server-chosen id that no other subscriber holds.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* security: accept only HS256 metadata subscription tokens
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* security: test HS384 metadata subscription token rejection
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* filer: never let a scoped subscriber replace another registration
The server-chosen id can be taken by another subscriber between choosing
and registering it. Registering scoped streams with the lowest epoch makes
addClient refuse them rather than replace the other registration.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* filer: send position markers for events a scoped token withholds
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* filer: test a scoped stream withholding a whole event batch
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* filer: filter SendMsg on scoped metadata streams
The scoped wrappers only overrode Send, so a SendMsg on the embedded
stream would reach the client unfiltered. Route it through the same
filter and refuse any other message type. Ported from #12790.
Co-authored-by: mindaeon <30555587+mindaeon@users.noreply.github.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* filer: test a scoped subscription through the real subscribe loop
Run SubscribeLocalMetadata with a scoped token over a flushed backlog,
the in-memory tail and live events: only in-scope events arrive, sibling,
"..", encoded and cross-boundary rename paths are dropped, and no log
file references are sent although the client asks for them. Without the
subscribe key every event still arrives. Ported from #12790.
Co-authored-by: mindaeon <30555587+mindaeon@users.noreply.github.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* filer: test that the HTTP API refuses metadata subscription tokens
GET, HEAD, PUT, POST and DELETE all refuse a subscription token, also
when the subscribe key equals the read or write key. Ported from #12790.
Co-authored-by: mindaeon <30555587+mindaeon@users.noreply.github.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: mindaeon <30555587+mindaeon@users.noreply.github.com>
* s3: list each part number once, its latest attempt
Every UploadPart writes its own NNNN_<uuid>.part entry, so a part number
uploaded twice listed twice, superseded ETag included, and a max-parts page
could hold only the stale attempt. ListParts now groups the entries by part
number, reports the attempt sortEntriesByLatestChunk ranks first (the same
order completion uses), and pages over distinct part numbers, reading past a
page boundary that splits one part number's attempts.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3: test ListParts over repeated part attempts
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3: rank an empty part attempt by commit stamp or mtime
An empty UploadPart body stores no chunks, so sortEntriesByLatestChunk gave
it time zero and an empty re-upload always lost to the attempt it replaced.
Attempts now rank by the filer's commit stamp when both carry one, else by
the later of the first chunk's write time and the entry mtime.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3: keep only the newest attempt per part while listing parts
listPartAttempts held every attempt of a part number until the page was
built, so a heavily retried part grew one ListParts request's memory past
what max-parts bounds. It now compares each entry against the newest seen
for its part number with compareLatestPartAttempt, the comparator
sortEntriesByLatestChunk sorts with.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3: order part attempts by commit stamp, then attempt time
Comparing commit stamps only when both attempts carried one made the order
non-transitive across stamped and unstamped attempts, so listing and
completion could pick different attempts depending on input order. The
commit stamp now leads (unstamped entries predate the field) and the
attempt time breaks ties.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3: list part 10000 after the four-digit part numbers
Part names sort as text, so 10000_<uuid>.part lands between 1000.part and
1000_<uuid>.part. A page could end on part 10000 and the next marker then
skipped parts 1000-9999, and a legacy 1000.part could be reported before
the newer attempt of part 1000 was read. List the four-digit part numbers
first and part 10000 in its own prefix listing.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3: test ListParts paging across part 10000
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
test/s3: build s3-fast-list with its data map draining the last page
The data map task reads the channel and then the list-task state. When
the list task sends its last page and completes between the two reads,
the map is dumped without that page and the tool exits 0, so a one-page
prefix comes back empty. Read the state before the channel.
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* telemetry: refresh the server module to the root module's dependency versions
The server module replaces seaweedfs with ../.., so root dependency
bumps raised prometheus/common, prometheus/procfs and x/sys past the
versions its go.mod listed, and go build refused to run.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* ci: run the telemetry integration test on root go.mod changes
The telemetry server module builds against the root module through a
replace directive, so a root dependency bump can leave its go.mod stale
without touching telemetry/.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
The versions chart keeps the 8 versions with the most clusters on the last
day and folds the rest into "other". A new release starts small, so 4.54
(282 clusters) was folded in while 4.53 shrank as clusters upgraded,
which showed as a dip across the named versions. The newest plain release
now always keeps its own band.
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* mount: release the meta cache lock between batches of a rebuild wipe
deleteFolderChildrenForRebuild held mc's write lock while it listed every
cached child of the directory and deleted them. Every metadata read in the
mount (FindEntry, IsNameFresh, isOversized) takes the read lock, and a
waiting writer blocks new readers, so while a large directory was wiped,
lookups and getattrs in unrelated directories queued behind the scan. The
scan gets slower as leveldb accumulates the deletion markers of earlier
wipes of the same directory, which happens quickly once lookups rebuild
expired directories.
Wipe rebuildWipeBatchSize (1024) children per lock hold and drop the lock
between batches. A child is judged pinned under the same lock that deletes
it, so a local-only create is kept whether it was pinned before the wipe
started or between batches. The listing that refills the directory already
ran outside the lock, so batching adds no interleaving the build did not
allow.
Add LevelDBStore.BatchDeleteEntries, the counterpart of BatchInsertEntries.
Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
* mount: bound how long a lookup waits on the directory rebuild it starts
Since #11268 a lookup whose parent's cache expired rebuilds the parent and
every expired ancestor up to the mount root before it answers, and every
lookup beneath them joins the same builds. With a slow listing (a large
directory, a busy filer, or the meta cache lock held by another rebuild) a
plain stat, an ls -l, or the parent touch after mkdir waited for all of it.
Wait at most lookupRebuildTimeout (2s), then fall back to one LookupEntry
RPC. Since #12206 the shared build does not run on the requester's context,
so it keeps going after the waiter gives up and later lookups find the
directory cached. A timed-out wait records a rebuild attempt, so lookups
during the cooldown do not queue on a build that is already known to be
slow.
Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
* mount: wait for the rebuild's listing in TestLookupDoesNotWaitOutSlowRebuild
The build runs on its own goroutine, so the lookup that stopped waiting
for it can return before the build's ListEntries reaches the filer. The
test read `started` without waiting right after the lookup returned, which
relied on an ordering nothing guarantees, and the ListEntries count taken
afterwards assumed the same. Wait for the listing to arrive, bounded by
the test's deadline.
Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
* mount: keep children inserted while a rebuild wipe runs
A deferred create inserts its placeholder before Create registers the
handle that pins it. The single-lock wipe made such an insert wait until
it finished, so the placeholder survived; a batch that runs in that gap
judged it unpinned and dropped it, and the rebuilt listing then omitted
the file until its flush.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* mount: drop a cached path that an event vacates during a rebuild wipe
A build replays only the events later than its listing's snapshot and
leaves the earlier ones to the listing, which starts after the wipe. With
the wipe in batches a file's whole life fits inside it: created, flushed,
then unlinked or renamed away. Both events are earlier than the snapshot
and are skipped, the listing has no such file, and the placeholder the
wipe kept stays in the published directory until the next rebuild.
While a directory's wipe runs, an event buffered for it now takes the
path it vacates out of the store at once, fenced by versions like any
other event. After the wipe the store holds what the listing put there,
and a removal earlier than the snapshot may still arrive, so from then on
the event is left to the replay as before.
Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
* mount: check which file a removal took away during a rebuild wipe
Two things the removal applied during a rebuild wipe got wrong.
A placeholder is unversioned, so the version gate lets every removal
through. One that arrived late, for an earlier file of the same name,
took out the placeholder of a file created since. The entry the event
took away is now compared with the cached one, by inode or, with no
inode to compare, by creation time. An event that only names the file
cannot tell and is taken at its word.
The entry was deleted without OP=MV, so taking one name of a hard-linked
file out of the cache also took one off the count its other names share.
A replayed event that vacates a path already deletes with OP=MV.
Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
* mount: keep a removal the rebuild wipe declined out of the replay
The wipe declines a removal when what is cached there is newer than the
event or is another file than the one it took away. The event stayed
buffered anyway. An unversioned listing clears the children's version
records and replays every buffered event, so the declined removal then
deleted the entry the wipe kept and the directory was published without
it.
A declined removal now buffers only the name it puts in place, so a
rename still lands on replay.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* mount: check a declined removal again when the build replays it
The previous commit dropped the removal half of an event the wipe
declined, judged by the file cached when the event arrived. Earlier
buffered events can replace that file during the replay: a listing
attempt that failed leaves its events buffered, so the retry's wipe may
see the first attempt's file while the replay puts a later one in place
first. The dropped delete then left that later file published.
The wipe now records which removals it declined, and the replay checks
each against the entry it finds at that point, replaying only the name
the event puts in place when that entry is still another file. A
removal declined by version is replayed as before.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* mount: keep a declined removal's invalidation off the placeholder's handle
After the replay the build invalidates every buffered event again,
raised to the snapshot, and it still sent the original of a removal the
replay rejected. A late delete of an earlier file then reached the
handle of the placeholder at that name: the placeholder is unversioned,
so the raised timestamp got past it and the handle was marked deleted.
The replay's decision on a declined removal now replaces the event in
the buffer, and the invalidations use what was replayed. It is made
before the snapshot check, so a removal the listing covered is judged
the same way.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* mount: keep an event's signatures on the halves a rename is split into
A declined removal replays as its rename's create half, and a rename across
a building directory is split in two. Both dropped the event's filer
signatures, so a handle stamped by that filer no longer fenced the replayed
invalidation by timestamp and could take older listed contents over newer
ones it had read.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* mount: drain invalidations around the declined-removal build check
Invalidations run on a background worker, so the baseline could miss the
removal's own and the final read could run before the build's arrived.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: MapleWhiteZZZ <18653045+MapleWhiteZZZ@users.noreply.github.com>
Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
* volume: fsync a batch delete that asks for it
BatchDeleteRequest gets an fsync field, and the Go and Rust volume
servers pass it to the needle delete, which already orders the
tombstone, .dat sync, publish and .idx sync for a durable delete.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* operation: pass an fsync flag to batch deletes
DeleteFileIdsWithLookupVolumeId and DeleteFileIdsAtOneVolumeServer take
an fsync argument and send it in the BatchDeleteRequest. Every caller
passes false for now.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* filer: fsync chunk deletes under an fsync path rule
DeleteChunks passes the path rule's fsync flag to the deletion queue.
The flag is kept with the pending record until the delete is
confirmed, so retries and re-queues keep it, and the batch and retry
loops send marked ids in a durable BatchDelete.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* filer: keep fsync marks in the deletion ledger
Marked ids go into the trailing parts of a manifest ledger, counted by
fsyncParts. Older filers read those parts as plain ids and still delete
them; a reload or a claimed foreign ledger restores the marks. A ledger
without marks keeps the single array form.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* filer: record a queued deletion before the delete loop can see it
queueDeletions published the ids to the hot queue before taking the
ledger lock, so a concurrent flush could delete an id before its fsync
mark landed and then forget the mark with the record.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* filer: fsync a folder delete's chunks by each child's path rule
A recursive delete collected its file children's chunks and resolved
the rule once for the folder, which misses a trailing-slash rule on
the folder itself and file-prefix rules. Each file child's own rule now
decides fsync; the folder rule still decides chunk deletion.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* filer: keep a newer fsync request when a plain delete finishes
An id re-queued with fsync while a plain delete of it was in flight
lost its mark when that delete succeeded, so the queued copy also ran
without fsync. A plain attempt now leaves a marked record for its
queued copy; a durable delete of an already-deleted needle still syncs
the tombstone.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* filer: delete a folder's plain child chunks without fsync
Children whose own rule turns fsync off were still deleted durably when
the folder's rule asked for fsync. Only the children marked by their
own rule now go out with fsync.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* filer: retry a durable chunk delete the volume server failed
A volume server rolls back a durable delete it cannot sync and answers
with a 5xx carrying the I/O error text, which the classifier took as
permanent, so the record and its fsync mark were dropped while the
needle stayed live. A 5xx answer to a durable delete is now retryable,
within the usual attempt limit.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* s3api: build request condition keys in one helper
Split the keys ExtractConditionValuesFromRequest reads from the request
itself into RequestConditionValues, keeping aws:SourceIp and the
bucket-listing keys in the method.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3api: evaluate request condition keys for STS sessions
The IAM integration context for STS and OIDC sessions carried only the
source IP, legacy user agent/referer keys and claims, so role and session
policy conditions on s3:x-amz-*, aws:SecureTransport, aws:CurrentTime,
s3:RequestObjectTag/* and the other request keys never matched: a Deny
on them never fired, and a negated Deny always did. Add the same request
keys the static identity path evaluates.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3api: report SigV4 presigned requests as REST-QUERY-STRING
s3:authType recognized only SigV2 query authentication, so a policy
condition on REST-QUERY-STRING never matched a SigV4 presigned URL.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* ci: cache the Snowflake suite's Maven dependencies under their own key
setup-java's maven cache key hashes the repository pom.xml files, which
other Java workflows share. The Snowflake job restored that cache, which
lacks the suite's plugins and dependencies, and never saved its own because
the key already existed. Every run downloaded the whole suite dependency
tree from Maven Central and failed when Central answered 429.
Key the cache on run.sh, which pins the suite revision.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* ci: fall back to Google's Maven Central mirror in the Snowflake job
A cold cache still resolves about 36 MB of plugins and dependencies, and
Maven gives up after three 429 retries within 30 s. Add the mirror as a
settings-profile repository; Maven tries it first and moves on to Central
for any artifact the mirror fails to serve.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* filer: mint read_auth for the data chunks behind manifest chunks
* filer: cover every chunk behind manifests in read_auth (data + nested), minted after resolution, and skip when read signing is off
* filer: pass the master client as manifest cache invalidator, and move the read_auth helpers below populateBatchReadAuth for the enterprise merge
* testutil: fetch pinned Maven Central jars with a mirror fallback
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* test: mount pinned Iceberg jars into the Spark catalog containers
Each test started a new Spark container that resolved the Iceberg runtime
and AWS bundle through Ivy from Maven Central, so a run downloaded them six
times and a single refused request left Spark retrying until the package
timed out.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* ci: cache the Iceberg jars for the Spark catalog tests
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* testutil: test the verified jar download
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* test: bound Maven jar downloads by one deadline shared across retries
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* test: give each Maven mirror attempt its own timeout within the download budget
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* s3api: accept an empty ListObjectsV2 continuation-token
An empty continuation-token lists from the start and is echoed back
empty, as S3 does; rejecting it broke s3-tests
test_bucket_listv2_continuationtoken_empty.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3api: leave the ListObjects V1 Prefix unencoded under encoding-type=url
S3 encodes the V1 Delimiter, markers and keys but echoes Prefix as sent,
and botocore, which always asks for url encoding, decodes the V1 Prefix
only for V2. Encoding it broke s3-tests test_bucket_list_prefix_unreadable.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3api: use a codespell-clean key in the versions encoding test
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3api: URL-encode a V1 Prefix that XML cannot carry
Leaving the V1 Prefix raw under encoding-type=url turned a control
character such as \x01 into U+FFFD in the response. Such a prefix is
URL-encoded again; prefixes that are valid XML text, including '\n',
stay as sent.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* master: export writable volumes per disk for each volume layout
The layout gauges only report how many volumes of a collection take
writes, not where they sit, so two writable volumes sharing one disk
cannot be seen or alerted on. The leader now exports
writable_volumes_per_disk with the layout labels plus data center, rack,
node and disk id, counting the replicas of the layout's writable volumes
on each disk (state=writable) and those of them that are crowded
(state=crowded). Series for disks that no longer hold a writable volume of
the layout, and for layouts that are gone, are deleted.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* master: reset the leader-only layout gauges on followers
The layout writable, crowded, under-replicated and per-disk gauges are only
set by the leader, so a master that lost leadership kept exporting the
values of its last term indefinitely, and dashboards that aggregate over
masters could show numbers days old. A master that is not the leader now
clears them on each periodic pass.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* grafana: chart disks holding several writable volumes of a collection
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* grafana: keep disks of different clusters apart in the per-disk panel
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* s3api: reject an empty ListObjectsV2 continuation-token
AWS answers continuation-token= with 400 InvalidArgument ("The continuation
token provided is incorrect"); SeaweedFS listed from the start of the bucket.
No token this server issues is empty, and raw-key and base64: tokens keep
resuming as before.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* s3api: reject an encoding-type other than url on listings
AWS answers any encoding-type other than url with 400 InvalidArgument
("Invalid Encoding Method specified in Request") on ListObjects,
ListObjectsV2, ListObjectVersions, ListMultipartUploads and ListParts;
SeaweedFS ignored the value and listed unencoded keys.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* s3api: apply encoding-type=url to ListObjectVersions
The handler accepted encoding-type=url but returned raw keys. It now
URL-encodes Key in Version and DeleteMarker entries, CommonPrefixes, Prefix,
Delimiter, KeyMarker and NextKeyMarker with the urlPathEscape form ListObjects
uses, and echoes EncodingType.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* s3api: URL-encode Prefix and Delimiter in ListObjects responses
Under encoding-type=url, ListObjects and ListObjectsV2 encoded keys,
CommonPrefixes, markers and StartAfter but echoed Prefix and Delimiter raw,
so a client decoding them once corrupted a prefix holding '%' or '+'.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* volume growth: spread writable volumes of a layout over servers and disks
Volume growth picked every server, and the volume server picked every disk,
independently: the server weighted by free slots, the disk by the lowest
total volume count. A grow request therefore stacks several writable volumes
of one collection on some disks while most disks get none (a Poisson spread:
with 160 replicas on 479 disks about 20 disks get two or three), and their
writes share those disks.
Data centers and racks are still picked weighted by free slots. Inside a
rack the server is now one of those holding the fewest writable volumes of
the layout per disk, weighted by free slots among them; this covers the main
server, same-rack replicas and replicas walked into other racks and data
centers. On the volume server the disk is the one with the fewest writable
volumes of the collection, then the lowest volume count as before.
A grow request registers each volume before placing the next, so a request
smaller than the number of disks lands on distinct disks.
Fixes#12746
* volume growth: scan the layout load once per grow request
Placing each volume rescanned all writable volumes of the layout under its
read lock. RecordAssign takes that lock exclusively on every assign, so with
thousands of writables (0.8 ms for 10000) every volume of a grow request
stalled assigns once more.
GrowByCountAndType now scans once and findAndGrow adds every volume it
creates to the counts. Per volume, placement costs about 130-160 us on 480
servers whatever the number of writables, against 120 us before the spread
change; the single scan is 0.17 ms for 2000 writables and 0.8 ms for 10000.
* volume growth: address review of the spread placement
- Count only disks that report volume slots in a server's load. An
unavailable disk, or one low on space with no volumes, is still listed in
DiskTags with no slots; counting it understated the load of a server with
failed disks and piled volumes onto its remaining ones.
- On the volume server, judge a volume full by its .dat size, which the
master compares with the limit, not by ContentSize.
- A server that refuses an allocation ranks last for the rest of the grow
request, and the request retries the volume up to 3 times. The least
loaded server is picked first deterministically, so without this one server
that cannot fit a preallocated volume stopped the layout from growing.
- Scan the layout load after warmup, so a grow request issued right after a
leader change sees the writable volumes the returning servers report.
* volume growth: keep the request-local placement load ahead of the request fields
The JSON request fields stay together at the end of VolumeGrowOption, and
the comment now says the load lives for the whole grow request.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* volume(rs): place a new volume on the disk with the fewest writable volumes of its collection
Store::add_volume picked the disk with the most free slots, so the Rust
volume server ignored the collection spread rule the Go server now applies.
find_free_location ranks disks by the writable volumes of the collection
first (not read-only, .dat below the volume size limit), then by free slots.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* volume growth: leave a refusing server out of every placement tier
A server that refused an allocation only ranked last among the servers of
its rack, so the weighted data center and rack picks could still choose its
rack on every retry while other racks had room. It now offers no free slots
for the rest of the grow request, which takes its slots out of the rack and
data center weights too. If only refusing servers are left, they are tried
again.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* volume growth: require the full same-rack count only in the main rack
The data center check asked every rack for SameRackCount+1 free servers,
but only the main rack holds the same-rack copies; each other rack takes one.
With a refused server left out of its rack, the check rejected a feasible
placement, the retry put the refused server back, and it was picked again.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Konstantin Lebedev <whitefox@mayflower.work>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* volume: fsync a needle delete that asks for it
A delete now takes fsync like a write: the tombstone is synced before the
needle map drops the needle, and the .idx row before the delete is
acknowledged. A failed .dat sync takes the tombstone back off the .dat.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* volume: honor fsync=true on DELETE and forward it to replicas
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* volume(rs): fsync a needle delete that asks for it
Mirrors the Go volume server: DELETE with fsync=true flushes the tombstone
before the index drops the needle and the .idx before the ack, and a failed
.dat flush takes the tombstone back off the .dat.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* volume: let a durable delete through on a full volume
A queued delete hit the batch worker's volume size check, which a
tombstone append is exempt from inline, so fsync=true deletes failed once
the volume reached its hard limit.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* volume: sync a durable delete of an already-deleted needle
An earlier delete without fsync may have dropped the needle with its
tombstone still unsynced, so the inline path syncs before acking anyway,
as the batch worker already does.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* volume: refuse a batch request that races the worker stopping
stopWorker can close the batch channel between startWorker handing it out
and the send, which panicked. The request now falls back to the inline path.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* volume(rs): sync a durable delete of an already-deleted needle
Mirrors the Go volume server: an earlier delete without fsync may have
dropped the needle with its tombstone still unsynced.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* volume(rs): run a durable delete on a blocking thread
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* volume: keep syncDelete's signature for existing callers
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* volume: refuse a write a delete cancelled before it appends
A write cancelled while still queued behind the volume locks used to
append, publish and then tombstone itself without a sync, after its
delete may already have been acknowledged as durable. A crash could
keep the row and lose the tombstone. Refuse it before the append.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* volume(rs): refuse a write a delete cancelled before it appends
Mirrors the Go change: a write cancelled while still queued behind the
volume locks no longer appends, publishes and tombstones itself without
a sync after its delete may already be durable.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* volume(rs): drop the write guards for a durable delete's flushes
The durable delete held the write lease and the io write guard across
both fsyncs, so reads of the volume waited on them. Append and publish
the tombstone under the guards and flush with only the io read side, as
write_synced_run and the Go path already do. do_delete_request keeps its
signature.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* volume: stop taking writes when a durable delete fails to index
The needle map drops the needle before it appends the .idx row, so after
a failed append a durable retry found nothing to delete and acked, and a
later write could bury the unindexed tombstone so a reload brought the
needle back. Fence the volume the way a failed durable write does.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* volume(rs): stop taking writes when a durable delete fails to index
Rust counterpart of the Go change: the needle map drops the needle
before it appends the .idx row, so a failed append left a durable retry
nothing to delete, and a later write could bury the unindexed tombstone.
publish_delete_request now fences the volume for a durable delete, as
publish_write does.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* volume: stop taking writes when any delete fails to index
A non-durable delete whose .idx append failed left the volume writable with
the needle already dropped from the map, so a later durable retry found
nothing to delete and acknowledged without the missing row.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* volume(rs): stop taking writes when any delete fails to index
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* volume: take the fsync flag on the existing delete functions
DeleteVolumeNeedle, deleteNeedle2 and syncDelete take fsync instead of
gaining *WithFsync variants.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* volume(rs): take the fsync flag on the existing delete functions
delete_volume_needle and delete_needle take fsync instead of gaining
*_with_fsync variants; the durable path and the deletable checks fold
back into delete_needle, and the cancelled-write check mirrors Go's
isCancelledWrite.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* test(s3): s3-listing-study integration suite (WIP)
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* test(s3): run the s3-listing-study tools against SeaweedFS (WIP)
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* test(s3): s3-listing-study deleted-prefix probe, fresh data dir per run
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* test(s3): listing study ports keep clear of every gRPC companion
free_ports.py compared only base ports, so a base port could land on
another server's base+10000 gRPC listener.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* test(s3): listing study ports stay below the ephemeral range
Base ports of 20000-55000 put both them and their gRPC companions inside
32768-60999, so an outgoing connection made while weed server starts could
take a port before the server that was given it binds.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* test(s3): listing study checks the order inside each listing page
parse_page sorted Contents and CommonPrefixes together before the probes
compared them, so keys returned out of order within one page passed. Check
each list's byte order first; the versions pager does the same.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* test(s3): listing study applies --only before running a probe
The filter was applied to the probe results, so a run limited to one tool
still sent every reference listing request and the deleted-prefix check.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* test(s3): listing study requires CommonPrefixes from shallow modes
The CommonPrefixes comparison ran only when a tool returned at least one
prefix, so a delimiter listing of the canary root that came back empty
passed. Compare them for every shallow mode except the two whose
normalizer keeps keys only.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* s3api: reject unknown elements in PutBucketNotificationConfiguration
xml.Unmarshal skips children that match no field, so a configuration
whose only child was an unmapped element parsed as empty and returned
success while nothing was stored.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* test/s3/tigris_compat: fail test-with-server when the server does not stop
A passing suite returned success even when stop-server gave up with the
server still running.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* test/s3: run tigrisdata-community s3-api-compat-tests in CI
Runs the suite, pinned to a commit, against weed mini through a small
provider patch, and fails on any test that is not in
expected_failures.txt or on a listed test that starts passing.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* test/s3/tigris_compat: prepare the suite again when the patch changes
The checkout was reused whenever it sat at the pinned commit, so an
edited seaweedfs-provider.patch, or a checkout whose patch failed to
apply, kept being built as it was. Record the commit and the patch's
hash once the patch applies, and prepare the checkout again when either
differs.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* test/s3/tigris_compat: keep the pid file of a server already running
start-server removed weed-server.pid before it found the port taken, so
a second start lost the pid of the server holding it and stop-server
could no longer stop that one. Remove the file only once the port is
free, and have test-with-server stop only a server it started.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* test/s3/tigris_compat: bound the S3 port probes
A listener that accepts but never answers held curl, and with it the
readiness loop and the cleanup after it, for good. Give each probe two
seconds.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* test/s3/tigris_compat: expect a listed test to end with its listed status
A listed test passed the check with any non-pass status, so a setup error
that skipped the call under test (RestoreObject's PutObject, say) went
unnoticed. Entries now expect fail unless they name another status; only
GetBucketAccelerateConfiguration, whose setup PUT is unsupported, expects
error.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* test/s3/tigris_compat: refuse to start while the recorded server still runs
A second start-server before the first server answered on the S3 port
passed the port probe and dropped the pid file of the live server, so
stop-server could no longer stop it.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* test/s3/tigris_compat: recognize the test server by its command line
start-server and stop-server took any process named weed at the recorded
pid for the test server, so a weed process that reused the pid would be
reported as running and killed by stop-server. Match the command line
start-server launches instead.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* test/s3/tigris_compat: recognize the test server by its start time
A weed mini started from another checkout has the same command line, so
a pid it reused still matched. Record the process start time next to the
pid at launch and require both before treating the process as the test
server.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* test/s3/tigris_compat: PutObjectTagging_VersionId passes now
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* test/s3/tigris_compat: keep the server records until the server exits
stop-server waited a fixed two seconds after SIGTERM and then removed
weed-server.pid and weed-server.started even if weed was still shutting
down, so a later stop-server could not find it and clean could delete
data it still used. Wait up to 30 seconds for the recorded server to exit
and keep the records, failing, while it still runs.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* test/s3/tigris_compat: start the server on an empty data directory
A data directory left by an earlier run on other ports made the master
freeze volume growth (topology id mismatch), and every PutObject failed.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* test/s3: run tigrisdata-community s3-api-compat-tests in CI
Runs the suite, pinned to a commit, against weed mini through a small
provider patch, and fails on any test that is not in
expected_failures.txt or on a listed test that starts passing.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* test/s3/tigris_compat: prepare the suite again when the patch changes
The checkout was reused whenever it sat at the pinned commit, so an
edited seaweedfs-provider.patch, or a checkout whose patch failed to
apply, kept being built as it was. Record the commit and the patch's
hash once the patch applies, and prepare the checkout again when either
differs.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* test/s3/tigris_compat: keep the pid file of a server already running
start-server removed weed-server.pid before it found the port taken, so
a second start lost the pid of the server holding it and stop-server
could no longer stop that one. Remove the file only once the port is
free, and have test-with-server stop only a server it started.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* test/s3/tigris_compat: bound the S3 port probes
A listener that accepts but never answers held curl, and with it the
readiness loop and the cleanup after it, for good. Give each probe two
seconds.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3api: accept PutBucketAccelerateConfiguration with Status=Suspended
GetBucketAccelerateConfiguration always reports Suspended, but the PUT
answered 501 for every request. Suspended is now accepted, Enabled
stays NotImplemented, and the accelerate subresource resolves to
s3:GetAccelerateConfiguration / s3:PutAccelerateConfiguration.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* s3api: accept an empty PutBucketNotificationConfiguration
GetBucketNotificationConfiguration always reports an empty
configuration, but the PUT answered 501, including for the empty
configuration clients send to clear notifications. An empty one is now
accepted; a topic, queue, Lambda or EventBridge destination stays
NotImplemented.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* s3api: reject accelerate and notification bodies with the wrong root
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* test/s3/tigris_compat: expect a listed test to end with its listed status
A listed test passed the check with any non-pass status, so a setup error
that skipped the call under test (RestoreObject's PutObject, say) went
unnoticed. Entries now expect fail unless they name another status; only
GetBucketAccelerateConfiguration, whose setup PUT is unsupported, expects
error.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* test/s3/tigris_compat: refuse to start while the recorded server still runs
A second start-server before the first server answered on the S3 port
passed the port probe and dropped the pid file of the live server, so
stop-server could no longer stop it.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* test/s3/tigris_compat: recognize the test server by its command line
start-server and stop-server took any process named weed at the recorded
pid for the test server, so a weed process that reused the pid would be
reported as running and killed by stop-server. Match the command line
start-server launches instead.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* test/s3/tigris_compat: recognize the test server by its start time
A weed mini started from another checkout has the same command line, so
a pid it reused still matched. Record the process start time next to the
pid at launch and require both before treating the process as the test
server.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* test/s3/tigris_compat: PutObjectTagging_VersionId passes now
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* test/s3/tigris_compat: keep the server records until the server exits
stop-server waited a fixed two seconds after SIGTERM and then removed
weed-server.pid and weed-server.started even if weed was still shutting
down, so a later stop-server could not find it and clean could delete
data it still used. Wait up to 30 seconds for the recorded server to exit
and keep the records, failing, while it still runs.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* test/s3/tigris_compat: start the server on an empty data directory
A data directory left by an earlier run on other ports made the master
freeze volume growth (topology id mismatch), and every PutObject failed.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* s3api: return x-amz-version-id from object tagging on versioned buckets
PutObjectTagging, GetObjectTagging and DeleteObjectTagging now report
the version they acted on, "null" for an object written before
versioning was enabled.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* s3api: set the tagging x-amz-version-id only on success
The header was set before the tag-conditioned policy recheck, so a
denied request carried the version id on its AccessDenied response.
Also test the "null" version and require distinct version ids.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* test/s3: run tigrisdata-community s3-api-compat-tests in CI
Runs the suite, pinned to a commit, against weed mini through a small
provider patch, and fails on any test that is not in
expected_failures.txt or on a listed test that starts passing.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* s3api: report no ownership controls after DeleteBucketOwnershipControls
GetBucketOwnershipControls reports BucketOwnerEnforced for a bucket that
never stored a setting, so it kept reporting it after the setting was
deleted. The delete now records that it happened, and GET answers
OwnershipControlsNotFoundError until ownership is put again. How
objects are written after the delete is unchanged.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* s3api: answer DeleteBucketOwnershipControls with 204 No Content
AWS answers a successful DeleteBucketOwnershipControls with 204 and no
body; this answered 200 with an empty OwnershipControls document.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* lance: per-table cleanup retention and on-demand purge of deleted rows
A vector deleted from an index stayed in storage until compaction
rewrote its fragment (only above 10% deletions) and the cleanup job's
168 h retention, set once per worker, aged out every version that
referenced it.
Let a table override retain_hours and min_versions in its Lance config
(seaweedfs.cleanup.retain_hours, seaweedfs.cleanup.min_versions), and
let it request a purge (seaweedfs.purge.requested). The cleanup job
proposes a table with a pending purge regardless of the version floor
and then rewrites every fragment with deletions, retrains the indices
(a remapped IVF_FLAT index still holds the deleted vector), keeps only
the latest version, and records the request as served. A version pinned
by a tag fails the purge and leaves the request pending.
The test scans every file of a local table for a deleted vector's key,
metadata and values, and finds none after the purge.
* lance: refuse to purge a sharded vector index
A retrain builds every index name over the whole table, so the purge
turned shards of one vector index (one name per fragment set) into
names that each covered every fragment, indexing every row once per
shard. Check before compacting and fail with the request left pending,
and retrain only the vector indices.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* lance: rebuild scalar indices in a purge
Compaction only remaps scalar indices, and a retrain leaves them alone,
so a purge reported success while a bitmap index still listed the
deleted row's value, and under stable row ids a BTree index still held
the deleted key. Build each scalar index again from the remaining rows
with the parameters it was built with.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* lance: purge through the cleanup job's retention policy
The purge built its own cleanup policy, so a deployment that has the
cleanup job keep more than the floor and the window (versions an
unleased reader may still be on) would have lost them to a purge. Clean
up through retention_policy, the one the job itself uses, with the
latest version kept.
Mark the request served only once the oldest version left is one the
rebuild wrote, and record the rebuild so a purge that has to wait for
its cleanup (a tag, or that policy) retries only the cleanup instead of
compacting and retraining the indices on every pass.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* vector: lease the version every list page reads
A list page read its version without a lease unless it was a segment
that read data and metadata twice. A purge removes every version but
the latest, and only a lease tag stops it, so a listing that opened the
version before the purge's compaction could lose its files mid-read.
Pin it as get_vectors and query_vectors do.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* lance: give the cleanup job a build's execution time
A purge compacts the table and retrains its indices inside a cleanup
job, but the job declared no runtime defaults, so the admin held it to
its 90 s default and could expire a purge mid-rebuild. Declare the same
budget the indices job uses for a retrain.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* mount: drop a peer-unlinked name only while its inode keeps another
Move the survivor check from followSurvivingLink into InodeToPath.RemoveAlias
so the check and the removal happen under one table lock.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* mount: follow a peer unlink to the surviving hard link for closed inodes
A peer deleting one name of a hard link only moved an open handle to the
surviving name. A closed inode kept the deleted name as its primary path, so
the next open by the kernel's nodeid failed with ENOENT.
Co-authored-by: skymazer <54621405+skymazer@users.noreply.github.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* mount: share the cached-entry freshness check of peer invalidations
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* mount: keep a name a local rename reclaimed before the peer unlink arrived
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* mount: keep a name a local rename or link gave the inode after an older peer delete
The cached-entry check only covers a cached parent. InodeToPath now records,
per hard-linked inode, the newest name a local rename or link gave it with
the filer stamp, and the closed-inode peer-delete path keeps that name when
the delete is older. The record lives in a side map dropped on Forget, so
InodeEntry keeps its size class.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* filer: sign rename stream responses with the filer signature
A rename response carries the stamp of the filer that applied it, so the mount
can tell which clock the stamp belongs to, as it does for create acks.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* mount: order a peer delete against a local rename or link only on one filer clock
The stamp of a local rename or link comes from the filer that applied it, and
a delete logged by another filer runs on a different clock. Keep the name only
when both carry the same nonzero filer signature; otherwise drop it as before.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* mount: drop the local-name stamp fence for uncached parents
Ordering a delayed peer delete against a local rename by filer stamp needs a
record per alias, a shared filer clock, and the open-handle path to follow;
that is a larger change than this fix. Back to the cached-entry check, which
covers a cached parent.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: skymazer <54621405+skymazer@users.noreply.github.com>
* mount: report opens for reading with TouchAccessTime (-accessReport)
An open answered from the metadata cache never reaches the filer, so a
filer that tracks access cannot see that a cached file is still being
read. With -accessReport, the mount sends one TouchAccessTime per open
for reading, with the open time, whether the open was answered from the
cache or not. Write-only opens are not reported.
Reports go out in the background and never hold up the open: a report
that does not fit in the queue is dropped, and drops and failures are
logged once a minute. A filer that answers Unimplemented pauses
reporting for 10 minutes.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* mount: address review of the access report
- Drop reports still queued when reporting pauses; the filer that answered
Unimplemented would answer them the same way.
- Bound each filer attempt to 2s, so a filer that accepts the connection
but never answers leaves time for the next one, and keep using the filer
that answered.
- Fail over only when a filer cannot be reached. Any answer, Unimplemented
included, ends the walk.
- Report an O_CREAT open of an existing file with read access, which comes
through Create instead of Open. A new file has nothing to read and is not
reported.
- Stop the reporter on unmount: stop ends the workers and cancels the
report in flight.
- Say in the flag help that it has no effect unless the filer implements
TouchAccessTime.
- Tests no longer wait a fixed time to see that nothing else was sent; they
look at the queue of a reporter that is not started.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* mount: fail an access report over only when the filer did not answer
A status the filer sent ends the walk even when its message contains a
word the transient classifier matches, so an Unimplemented answer still
pauses reporting and a rejected request is not resent elsewhere.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* mount: keep a filer selection made while an access report failed over
A report that reached a later filer moves the mount there only if the
selection it started from is still current, so a slower report cannot
undo another call's failover.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* mount: share an access report's time with the filers not yet tried
Each attempt is capped by an equal share of the report time left, so a
row of stalled filers cannot use up the report before a healthy one later
in the list is tried.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* mount: report the path an open for reading came through
A create that opens an existing hard link reports the name it was
given rather than the inode's first known name. Open reads the inode's
path from the inode table rather than the handle's saved path, which
other handle operations may rewrite concurrently.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* mount: check the access report stop and failover tests without races
The post-stop report is checked on a reporter without workers, so no
worker can take an accepted report before the check, and the failover
test asserts that the stalled filer was tried first.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* pb: keep the shared channel when the peer answers with its own status
gRPC never raises InvalidArgument, NotFound, AlreadyExists,
FailedPrecondition, OutOfRange or DataLoss itself, so such a status is
the peer's answer. It fell through to the string matching, and a message
naming a path like /transport closed the cached channel and cancelled
every other call on it.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* mount: say which filer selection an access report keeps
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* mount: bound the in-flight call in the rejected-report channel test
The call held in flight went through touchAccessTime, whose 2 s attempt
cap could expire on a slow runner and fail the test on a healthy channel.
It now uses callFiler with the test deadline. The wait for it to reach
the filer also ends if it returns early or the deadline passes, and the
blocked handler is released on every exit path.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* mount: report each file at most once per -accessReport.interval
Every read open used to send a TouchAccessTime, adding a filer metadata
call per open. A path reported within the interval (default 1h) is now
skipped; the last report times live in a bounded LRU. 0 keeps reporting
every open.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* mount: report an open again once the file was modified since its last report
Within -accessReport.interval a read open was skipped even after a write,
so the filer kept an atime older than the mtime until the interval ran
out. relatime records that first read after a write; do the same by
comparing the entry's mtime with the last report.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* mount: keep the report time apart from the mtime it saw
Recording max(now, mtime) as the last report let a file dated a day ahead
suppress reports for a day plus the interval. Record when the report was
sent and the mtime the entry had, and treat any change of that mtime as a
modification, so neither clock can stretch or shorten the interval.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: MapleWhiteZZZ <18653045+MapleWhiteZZZ@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
* shell: volume.check.disk skips replicas whose live counts match
FileCount counts every write and DeleteCount every delete or overwrite
until a vacuum drops them, so a replica vacuumed while its peer was not
keeps a different FileCount forever and was re-checked on every run
although both hold the same live needles. Only treat counts as differing
when the live counts (FileCount - DeleteCount) differ too.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* shell: volume.check.disk skips equal live counts only across compaction revisions
Equal live counts at the same compaction revision mean a missed write
and a missed delete, not a vacuum-state difference, so keep checking
those pairs.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* shell: volume.check.disk real-time recheck ignores heartbeat revisions
VolumeStatus counts are newer than the heartbeat's compaction revisions,
so a vacuum in between could make a pair at the same revision look
vacuum-skewed. Apply the old count rules there.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* shell: volume.check.disk skips on live counts only when no repair could follow
Equal live counts across compaction revisions can still hide a missed
write and a missed delete. Keep checking such pairs when -syncDeleted
could propagate the delete, or when -resurrectMissingNeedles could copy
a needle into a never-vacuumed replica.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* s3vectors: enforce AWS metadata, key and filter limits
PutVectors accepted vector metadata of any size and key count, bounded
only by the request body limit, and keys of any length; QueryVectors
accepted filters of any size. Apply the AWS S3 Vectors limits
(https://docs.aws.amazon.com/AmazonS3/latest/userguide/s3-vectors-limitations.html),
taking the ENHANCED index values where the modes differ, since this
implementation prefilters and takes no indexMode:
- 100 KB total metadata per vector, which also bounds a single
non-filterable field;
- 2 KB filterable metadata per vector;
- 50 metadata keys per vector;
- 1024-character vector keys;
- at most 10 nonFilterableMetadataKeys of 1 to 63 characters at
CreateIndex;
- 100 filter constraints per QueryVectors request.
A violation is a ValidationException before the engine is called.
* s3vectors: accept metadata whose JSON is exactly at the byte limits
metadataEntrySize counts a trailing comma for every member, so starting
from both braces overcounted by one and rejected metadata that encodes to
exactly 2 KB filterable or 100 KB total.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3vectors: rewrap the metadata limits header comment
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* kms/openbao: refresh the AppRole token before it expires
The provider logged in with AppRole once, when it was created, and
never renewed the token. After the token's TTL every SSE-KMS upload,
and every read of an SSE-KMS object whose data key was not cached,
failed with "permission denied" (HTTP 500 to the client) until the S3
server was restarted.
With AppRole authentication the provider now logs in again in the
background once two thirds of the token's TTL have passed, and retries
a failed login every 5 s. Each login is bounded by the client's
timeout, or by 30 s when request_timeout is not positive. Close stops
the refresh. A static token, or an AppRole token without a TTL, is left
as it is.
* kms: keep the OpenBao token refresh across configuration reloads
- AddKMSProvider and InitializeGlobalKMS create a provider instance of
their own with the new NewProvider instead of the registry's cached
one: a reload closed the cached instance and then got it back, which
now stopped its token refresh. The new instance is created before the
old one is closed, so a configuration that fails leaves the current
provider working
- openbao: time each refresh from the start of the login that issued
the token, since its TTL runs from then rather than from the response
- tests: a reload gives a new instance and closes the old one, a failed
reload keeps the old one; the refresh survives a reload; a slow login
response
* kms: create a replacement provider outside the manager lock
AddKMSProvider ran the provider factory, which for OpenBao is a login
that may wait for its timeout, while holding the manager lock, so every
lookup, also of unrelated providers, waited for it. The new provider is
now created before the lock is taken; the lock covers only the swap, and
the replaced provider is closed after it. Test: a lookup while another
provider is being created.
PutObject rejected a Cache-Control value it could not parse with
InvalidDigest, whose message is about Content-MD5, even when the request
sent no Content-MD5. CreateMultipartUpload already answers the same parse
error with InvalidRequest; PutObject now does too.
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* operation: reassign an upload after a proven-expired volume JWT
* seaweed-volume: mark an expired write JWT refused before append
Mirror the Go volume server: a multipart POST whose write JWT is signed
with the current key, names this fid and fails only on exp gets
X-Seaweed-Write-Outcome: expired-jwt-before-append on its 401, so the
uploader reassigns instead of failing. Both checks use one guard
snapshot, so a key rotation between them cannot drop the marker.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* operation: keep same-FID retry for filer-proxy expiries, snapshot the write signing key, and make the RetryForever budget test fail fast
* operation: re-mint the filer credential before a proxy same-FID expiry retry
* operation: make a proxy marked-expiry terminal in the outer retry too, not only the inner same-FID loop
---------
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* mount: compare permissions against local ids without mapping them again
Access, open and create (without -defaultPermissions) ran FilerToLocal on
entries that maybeReadEntry and maybeLoadEntry already return in local ids.
Under a uid/gid map where a local id is also a mapped filer id, the owner
was mapped twice and the check refused the file's own owner.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* mount: cache a deferred create's placeholder in filer ids
The meta cache stores filer uid/gid and maps them to local ids on every
read, but createRegularFile cached the deferred or writeback placeholder
with the caller's local ids. The second mapping in the permission checks
used to cancel that out under a swapping id map; without it the owner of
a placeholder was refused. Cache a mapped clone and keep the handle's
entry in local ids.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* mount: test that a deferred create returns its entry in local ids
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* filer: stop following a peer that stays unreachable
* filer: re-track an answering peer only while it is still followed
A signature read that succeeds just before the master removes the peer
called initPeerWatermark after the removal, clearing its mark. The loop
then exits, so the watermark it re-tracked never advances or drops and
pins aggregated readers at the settled horizon.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* filer: forget a peer's unreachable clock when the master removes it
Only a successful connection cleared the clock, so every departed address
that once failed stayed in the map for the filer's lifetime, and a re-added
peer inherited the old outage. The removal now drops it, and the clock is
guarded by peerChansLock so the removed peer's exiting loop cannot start a
new one.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* filer: report the unreachable mark only when it is set
Once peerResumeRetention forgets the mark of a peer that stays
unreachable, markPeerWatermarkRemoved no longer marks it (its watermarks
are gone), but notePeerUnreachable still saw no mark and logged
"treating it as removed" on every retry. markPeerWatermarkRemoved now
reports whether it set a new mark.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* test: fix the unreachable-peer subscriber's start before the baseline write
The subscriber goroutine picked its start time itself, so a baseline write
that finished first fell before the subscription and the test failed after
20 seconds with correct delivery.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* mount: answer an interrupted read with EINTR, not EIO
The kernel's FUSE contract (Documentation/filesystems/fuse/fuse.rst,
"Interrupting filesystem operations") lets a filesystem either ignore an
INTERRUPT or honor it by replying EINTR to the original request. Read did
neither: the FUSE cancel channel cancels the read's context, the chunk
fetch fails with "context canceled", and every error became EIO.
Interrupts come from any signal the reader handles. Go's runtime sends
SIGURG for goroutine preemption, so Go programs reading under load hit
this on healthy data: on a 128-thread 64 KiB random-read test about 250
of 5 million reads failed with EIO, and none with GODEBUG=asyncpreemptoff=1.
Read now returns EINTR when its own context was cancelled, the way
dirReadStatus already does for directory reads and lookups do since
#11717. Go's os.File retries EINTR, so those readers see no error. A
cancelled read also skips the entry refresh retry, and neither read log
line reports an interrupt as a warning or error. Failures without an
interrupt stay EIO.
Tests: TestReadInterruptedReturnsEINTR (fails on the old code with EIO),
TestReadFailureWithoutInterruptReturnsEIO; weed/mount/... passes.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* mount: require the failed-read test to reach the filer
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* mount: skip the read retry once the request is interrupted
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
* mount: move an open file's read view to the chunks a merge wrote
When a flush finds that a file's chunks take more than twice its size,
maybeMergeChunks reads the file back and writes it again as new chunks.
The commit that publishes them makes the filer delete the chunks they
replaced. The flush put the new chunks into the handle's entry but left
its read view on the old ones, so every descriptor that kept the file
open went on reading the replaced chunks. Those read back until their
volume is vacuumed. After that a read first retries the fetch of the
missing needle for about 13 s; then refreshEntryForRead installs the
filer's entry and the read is retried. The refresh skips a handle with
writes it has not flushed, and such a read returns EIO instead.
Files that are kept open and rewritten in place, such as PostgreSQL
data files and SQLite databases, are the ones whose chunks bloat, so
they are the ones that hit this.
installRewrittenChunks puts the merge's chunks into the entry and the
read view together, under the entry lock. Chunks attached while the
merge ran were not part of it and stay. Flushes without a merge are
unchanged.
TestMergeMovesReadViewOffReplacedChunks covers the handle.
TestReadOpenHandleAfterChunkMerge runs it on a real mount, with and
without an unflushed write; for it, the fuse_integration framework
starts mini on a free master port and exposes the address.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* mount: drain uploads before a merge replaces the chunks
Uploads attach their chunks without the handle lock, first to the entry
and then to the read view, and writes that landed after doFlush drained
the pages may still be uploading when the flush takes its lock. An
upload that attached during the merge could keep a chunk older than
data the merge already holds, or put a chunk the merge replaced back
into the read view. Either outranks the merge's zero-timestamp chunk.
When the chunks are bloated enough to merge, the flush now drains the
pages again while its lock holds writes off, and takes the chunks again.
FlushAll returns only after every upload has attached to both the entry
and the read view, so nothing attaches during the merge.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* mount: leave a pre-merge upload failure to the next flush
The drain before a merge returned an upload failure through the
metadata flush. doFlush then retried it as a filer error and answered
EIO, where the page flush answers a full disk with ENOSPC. The drain is
there only to wait for the uploads: a failure stays with the pages, and
the next flush reports it as a write error, as it would without a merge.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* mount: keep chunks attached while a flush installs its compacted list
A write that lands after doFlush drained the pages can still be uploading
when flushMetadataToFiler takes its chunk snapshot. Its upload attaches the
chunk without the handle lock, and the plain flush then replaced the
entry's chunks with the list compacted from the snapshot, dropping it from
the handle and from every later commit. Keep chunks attached after the
snapshot, as the merge path already does.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* mount: fail a merge's flush on a failed drain
A failed upload found by the drain before a merge was only logged, and
the flush went on to publish a size covering bytes that no chunk stores.
It now fails the flush as a pageUploadError before anything is
published; doFlush maps it with writeErrorToFuseStatus, as it does its
own FlushData failure, and does not retry it as a filer error.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* mount: clone a flush's request entry under the entry lock
The request snapshot cloned the live entry after GetEntry had released
its read lock. Uploads attach chunks without the handle lock, so an
AppendChunks that landed during the clone raced it on the Chunks field.
Clone inside UpdateEntry, as content flush recovery does.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* mount: leave a merge to a later flush while uploads are pending
Before a merge the flush drained the pages while it held the handle and
publication locks. A writeback upload retries a transient storage error
until it succeeds, so that drain could hold both locks for as long as
storage stayed down, stalling reads, writes and stat on the handle.
The flush now checks for pending uploads and, if any, publishes the
compacted chunks and leaves the merge to a later flush. With none
pending the lock still holds writes off, so nothing attaches during the
merge, and a failed upload still fails the flush as a write error.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* mount: keep room for pending uploads when a flush merges onto a moved entry
A flush that leaves the chunk merge to a later flush publishes while
uploads are still pending, and writes that landed after the drain can be
uploading on any flush. Merging a peer's inline content or a shrink into
such a publication lets the upload attach its chunk next to the inline
bytes, where reads never reach it. Tell createContentEntry when uploads
are pending so the merge keeps room for them.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* mount: test chunk merges through the filer's chunk proxy, also under load
The tests serve the filer's chunk proxy, and like the filer the fake
deletes the chunks each commit replaces, so a read view still naming
them fails:
- TestFlushMergeThroughChunkProxy/Quiet: the flush merges, and reads
through the open handle return the file from the merged chunk alone.
- TestFlushMergeThroughChunkProxy/UploadInFlight: an older page is still
uploading while a newer page of the same bytes has attached. The flush
leaves the merge, reads return the newer page, and the next flush
merges.
- TestFlushMergeRacesWithWritesAndReads: four writers, three flushers
and a reader on one handle, with uploads delayed at random. The
writers pause now and then, so 14 to 18 flushes a run merge and the
others leave the merge to a later one. Every read must succeed within
2 s, and at the end the handle and the filer must both hold exactly
what was written.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* mount: let a held upload go when UploadInFlight fails
The subtest's proxy holds the older page's upload until the test
releases it, and only the passing path did. A failure in between left
the handler blocked, so the proxy's cleanup waited on it and the failure
surfaced as a suite timeout. Release it from a cleanup as well, once,
and bound the waits for the upload and the drains.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
* mount: ask the filer again when a deferred create is published mid-lookup
A program that writes a file under a temporary name and renames it while
the file is still open (safetensors save_file does, through Rust
tempfile's persist) looks the temporary name up before the rename. When
the parent directory's listing is not cached, that lookup goes to the
filer. If a FLUSH publishes the deferred create while the lookup is in
flight (a forked child closing its inherited descriptor sends one), the
filer can answer not-found from a read taken before the create
committed. By the time the answer arrives the flush has cleared the
handle's dirty flag, so lookupEntry finds nothing local to explain the
answer, logs "possible coherence bug" and returns ENOENT, and the rename
fails although the file exists.
Note before sending the request whether the inode had unpublished
metadata (a dirty handle or an async flush in flight). On not-found, if
it had and has since been published, ask the filer once more: the filer
acknowledged the publish before the second request is sent. A second
not-found stands, so a file another mount removed is not brought back.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* mount: judge a placeholder's not-found by one look at its base
Review of the previous commit found two problems. The retry check and the
not-found fallback read the handle separately, so a flush that finished
between the two reads still turned the answer into ENOENT. And the
pre-request read of dirtyMetadata, a plain bool that Write and the flush
set under the handle lock, raced with them on every lookup that reached
the filer.
Note instead whether the handle is a local placeholder: a deferred create
has no filer base (baseEntry) until a flush publishes it. A not-found for
a placeholder is judged by one atomic look at that base. With none, the
filer cannot have the file yet and the handle answers. With a base, the
filer acknowledged the publish, so ask once more. dirtyMetadata is not
read for this, and the not-found fallback keeps the existing check, with
the placeholder decision added.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* mount: answer a placeholder from its own handle; a second miss is final
Follow-ups from review of the previous commit:
- Pick the placeholder after the async-flush wait, from the path's
current mapping.
- A placeholder that still has no base answers from the observed handle
itself, instead of through the not-found fallback, which re-resolved
the path and could miss once the handle was released and the cache
entry evicted.
- A not-found from the second request is final. The flush installs the
base before it clears dirtyMetadata, and keeps it set when a write
landed meanwhile, so the fallback could return a file another mount had
removed.
- Test gates release on cleanup, so a failing test leaves no handler
blocked.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* mount: read a placeholder's entry the way GetAttr does
The previous commit answered a not-found for a placeholder from the
handle's live entry under entryLock. Neither of the entry's writers takes
that lock: Write and the metadata flush change size, times and chunks
under the handle lock, and async upload workers append chunks under the
LockedEntry lock, so the lookup could read an entry while it changed.
Answer with the copy the create cached when it is still this file's
(same inode), as the not-found fallback already prefers. Otherwise copy
the handle's entry under both writers' locks, in GetAttr's order (handle
lock shared, then the LockedEntry lock), and convert the copy. No caller
of lookupEntry holds a handle lock.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* mount: cache a deferred create's placeholder in filer ids
The meta cache stores filer uid/gid and maps them to local ids on every
read, but createRegularFile cached the placeholder with the caller's local
ids. Under a uid/gid map whose local id is also a mapped filer id, a lookup
or stat answered from the placeholder reported the wrong owner. Cache a
mapped clone and keep the handle's entry in local ids.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* mount: assert the placeholder is cached before checking its owner
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* mount: check the snapshot owner after evicting the cached placeholder
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
* helm: set publishNotReadyAddresses on the volume headless service
* fix(helm): separate volume DNS discovery from ingress traffic
* test(helm): cover volume DNS discovery and ingress routing
* fix(helm): rename the volume routing Service so volumes.client cannot collide
* docs(helm): name the ready-only Service for explicit volume backendRefs
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* fix(helm): fail when the volume routing Service name collides after truncation
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* docs(helm): point explicit volume backendRefs at the rendered client-volume name
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* volume: remove files a failed VolumeEcShardsCopy wrote
The shard files were copied before the .ecx, so a source holding a
0-byte index failed the request only after its shards had landed, and
nothing removed them: every retried move stranded another unmounted
shard on a new disk.
Copy the .ecx first, staged beside its target, so a bad index fails
before any shard lands; rename it into place only after the shards and
sidecars have copied, so a failed copy never truncates or replaces an
.ecx the destination already has. On any error, remove exactly the
files this call created; files already on the disk (an existing
holder's shards, index or journal) are left alone. Go and Rust volume
servers behave the same.
Claude-Session: https://claude.ai/code/session_011UpvafcfUuC2mLxNAXLQJi
* ec_balance: skip EC volumes whose every holder reports an empty .ecx
A shard copy refuses a 0-byte .ecx, so detection proposed the same
failing move of such a volume every cycle. Leave volumes with no
holder reporting a needle out of planning; their shards still count
toward disk occupancy.
Claude-Session: https://claude.ai/code/session_011UpvafcfUuC2mLxNAXLQJi
* volume: serialize VolumeEcShardsCopy per volume
Overlapping copies of one vid shared the staged .ecx, and a failed copy's
cleanup removed shards that a concurrent copy had already committed and
mounted. Hold a per-vid EC lifecycle lock across the whole copy and its
cleanup, named as in the enterprise build (Store.EcLifecycleLock /
ec_lifecycle_lock).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* volume: overwrite an existing .ecx in place when committing a copied index
A mounted EcVolume reads its index through an open handle, so renaming
the staged copy over it left the reader on the replaced inode: tombstones
the source had folded into its .ecx never reached the mounted holder.
An index already on the disk is now overwritten in place, as before the
staging change; a fresh destination still gets the rename. Go and Rust.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* volume: stage replacements of shards a VolumeEcShardsCopy finds on disk
The shard transfer opened an existing destination shard with O_TRUNC and
removed it on a stream or write error, so a failed retry deleted a shard
this disk already held. A shard already present is now copied to
<shard>.copying and renamed into place only after every other file has
copied; a mounted copy is reopened so it serves the new file. Fresh
shards still write in place and are removed on failure. Go and Rust.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* volume: stage the .vif and .ecsum a VolumeEcShardsCopy brings until commit
The sidecars landed as soon as they transferred, ahead of the deferred
.ecx commit, so a failure after them left a new .vif beside the index
the failed copy kept. They are now staged as .copying and renamed into
place with the shards and index once every transfer has succeeded; a
source without one still stages nothing. Go and Rust.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* volume: run a Rust VolumeEcShardsCopy to completion when its caller goes away
tonic drops a cancelled handler future at its current await, which
released the EC lifecycle lock while file operations the copy had started
(a cleanup remove_file on the blocking pool, a create or rename) could
still land; a retry taking the lock could then lose its files to them.
The copy now runs in a spawned task that owns the lock, as Go's handler
runs to completion.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* volume: fail the sidecar copy tests with an unreadable source .ecsum
A directory at the source path reads as "not found" on volume servers
that skip non-regular EC files, which turned the failure into a no-op.
A mode-000 file fails the transfer on every implementation; root and
Windows, which can read it, skip the test.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* volume: copy each shard a VolumeEcShardsCopy lists once
A shard listed twice and already held here was staged twice; the first
commit rename consumed the staged file, so the second failed the request
after the shard was replaced, and every retry failed the same way.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* volume: wait out an in-flight VolumeEcShardsCopy before mounting its shards
A shard the copy had already landed could be mounted while the copy was
still running; the copy's failure cleanup then removed the file but left
the shard registered and announced. MountEcShards takes the per-volume
lifecycle lock, as the enterprise build already does, and the Rust mount
handler takes the same lock.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* volume: fail a VolumeEcShardsCopy whose replaced mounted shard cannot be reopened
The copy reported success while the mounted shard kept serving the
replaced file. The error lets the caller retry the copy instead of
treating the shard as delivered.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* volume: check which transfer fails in the EC copy cleanup tests
An earlier failure, before any shard or sidecar landed, would leave the
destination unchanged and pass every assertion without exercising the
cleanup. The sidecar test also skips on Windows, where a mode-000 file
stays readable.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* volume: fail the sidecar transfer in the EC copy test by injection
A mode-000 file is still readable to root and on Windows, so the test
skipped there. The source now returns an error for the .ecsum request
only, and the test checks that it served .vif before that failure.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* volume: take the EC lifecycle lock for a blanket full teardown
A teardown acknowledged while a detached copy of the same volume was
still waiting on its source let that copy recreate the index and shards
afterwards. Go and Rust now wait for the copy, as the enterprise build
does.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* volume: keep the request id on a Rust EC shard copy
The copy now runs in a spawned task, outside the gRPC layer's request id
scope, so its CopyFile calls to the source went out with a new id. Carry
the caller's id into the task.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* volume: unmount a replaced EC shard whose reopen fails
The failed copy left the mount reading the replaced file's inode, so the
destination kept serving the old shard. Unmount it, which also reports
the removal to the master, so a retry copies and mounts it again.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* volume: make the Rust EC copy tests fail for the right reason
The sidecar test now serves a directory as the source .ecsum, which root
cannot read either, so it no longer returns early there. The cancel test
holds the store until the handler's join reports the cancellation, so a
copy that finished before the abort can no longer pass it.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* volume: stage every shard a VolumeEcShardsCopy transfers
A shard missing here was written in place with a cleanup guard armed
before its transfer started, so a failed copy removed a shard that
ReceiveFile wrote meanwhile. Every shard is now staged, and only one
that did not exist when it is moved into place is removed on failure.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* volume: take the EC lifecycle lock for VolumeEcShardsRebuild
A rebuild folds the deletion journal into .ecx. Run during a copy of the
same volume, the copy's index then replaced the folded tombstones. Go
and Rust take the lock the enterprise build already takes here.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* volume: check the staged .vif bytes in the EC sidecar failure test
A source without the optional .vif also returns success, so the served
extension list could not prove the transfer. The test now reads the
staged .vif when the .ecsum request fails and compares it with the
source's.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* volume: unmount a failed reopen's shard on the copy's disk only
UnmountEcShards walks every disk, so a reopen failure on the destination
also unmounted a healthy copy of the same shard on a sibling disk. The
copy now unloads the shard from its own disk and reports that disk's
removal.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* volume: take the EC lifecycle lock for an ordinary EC shard delete
Shards a copy has staged are invisible to the delete's last-shard check,
so deleting the only canonical shard removed the index the copy was
about to commit its shards against. Go and Rust deletes now wait for an
in-flight copy of the same volume.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* volume: leave a mounted shard's copied file in place on copy failure
A shard mounted on the copy's disk whose file was missing got the copied
file reopened into its mount, then that file was unlinked by a later
failure's cleanup while the mount kept serving it. Only an unmounted
shard's new file is removed now.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* volume: skip repeated shard ids with a seen set
Scanning the request prefix for each id made a request with many
repeats quadratic while the copy held the volume's lifecycle lock.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* volume: make the Go EC copy and lock tests observe their effects
The fake source counts requests per extension, so the duplicate-id test
proves .ec05 is fetched once, and returns fixture errors instead of
failing the test from its handler goroutine. The staged .vif reaches
the test through a channel. The mount and teardown lock tests seed real
files and check that nothing happened while the lock was held and that
the operation took effect after it was released.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* volume: poll the handler directly in the Rust EC lifecycle lock tests
A spawned handler that had not run when the 200 ms timer fired passed
the blocking check without ever reaching the lock. The tests now poll
the pinned handler future themselves, which guarantees it is waiting on
the lock, and cover the ordinary shard delete as well.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* volume: observe the lifecycle-lock waiter in EC copy lock tests
The mount, teardown, shard-delete and rebuild tests slept 200 ms while
holding the lock, so a handler scheduled only after the unlock passed
without ever taking it. Wait until the handler's goroutine is blocked on
the mutex instead; each test now fails within its timeout if the handler
skips the lock.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* volume: check the shard is staged before the failure-path write in a copy test
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* volume: recover a missing EC index under the volume's lifecycle lock
Missing-index recovery fetched the index and then ran a store-wide mount
of every orphan shard, without the per-volume lifecycle lock. It could
mount shards an in-flight VolumeEcShardsCopy of another volume had just
renamed in, and that copy's failure cleanup then unlinked a mounted shard.
Each recovered volume now holds its lifecycle lock across a re-check, the
fetch and a mount restricted to that volume.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* volume(rs): recover a missing EC index under the volume's lifecycle lock
Mirror of the Go change: recover_missing_ec_indexes takes the volume's
ec_lifecycle_lock, re-checks that the index is still missing, fetches it,
and mounts only that volume's shards through mount_recovered_ec_shards_only,
so an in-flight VolumeEcShardsCopy of another volume never has its
renamed-in shards mounted from under it.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* volume: mount a volume whose index a copy installed during recovery
Recovery re-checks under the lifecycle lock that the index is still
missing. When a copy installed it in the meantime, the re-check skipped
the mount too, and the volume's other shards stayed unmounted because
later scans no longer list it. Skip only the fetch and still mount the
volume's orphan shards under the lock, in Go and Rust.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* volume: fetch a recovered EC index into the disk the locked re-check found
The re-check under the lifecycle lock only kept a yes/no answer, so the
fetch still wrote to the destination from the first scan, which may be a
disk that has since gone away. Fetch into the record the re-check returned.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* volume: mount a copy-installed EC index even when no peer is known
Recovery skipped the locked step when the master was unreachable, the
lookup failed or no peer was listed. A copy that installed the index after
discovery then left the volume's other shards unmounted, and later scans
no longer list it. Always run the locked re-check, and give up for lack of
peers only when the index is still missing.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* volume: grow the stack buffer before scanning for a lock waiter
A full buffer can cut off the handler goroutine's stack, so the test would
report a handler that is waiting as one that never waited.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* volume(rs): fetch a recovered EC index into the disk the locked re-check found
Mirror of the Go change: fetch into the record the re-check under the
lifecycle lock returned instead of the destination from the first scan.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* volume(rs): mount a copy-installed EC index even when no peer is known
Mirror of the Go change: a failed lookup or an empty peer list no longer
skips the locked re-check, so a volume whose index a copy installed after
discovery still gets its orphan shards mounted.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* volume(rs): run missing-index recovery to completion when the mount RPC is cancelled
Dropping the mount handler mid-recovery released the volume's lifecycle
lock while the .ecj merge it had started on a blocking thread was still
landing, so a teardown could remove the volume before that late write.
Run recovery on its own task, as the copy handler does, so the lock is
held until it finishes; Go's handler already runs it to completion.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* volume: skip other volumes before stat in a single-volume EC scan
Recovery re-checks and mounts each volume with its own scans, so a
request with many orphans repeated whole-store discovery once per volume.
The index-owner and orphan-shard scans now take the same volume filter
and drop other volumes' files by name, before the per-file stat, and the
locked re-check scans only its own volume.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* volume(rs): skip other volumes before stat in a single-volume EC scan
Mirror of the Go change: the index-owner and orphan-shard scans take the
volume filter and drop other volumes' files by name before the per-file
stat, so recovery's per-volume re-check and mount no longer repeat
whole-store discovery under the store lock for every orphan.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* volume(rs): announce shards recovered after the mount RPC was cancelled
Recovery now outlives a cancelled mount RPC, but the heartbeat nudge
stayed in the handler, so shards it mounted after the caller left waited
for the next full EC heartbeat. Nudge it from the recovery task, as Go's
loads announce each recovered shard.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* volume: match the collection as well as the volume id in the recovery re-check
Discovery keys orphans by (collection, vid), but the locked re-check
matched the vid only. Two collections with orphans of the same vid could
both resolve to the first record, so the other's index was never fetched.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* volume(rs): match the collection as well as the volume id in the recovery re-check
Mirror of the Go change.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* s3api: answer NoSuchBucket from GetBucketRequestPayment for a missing bucket
The handler wrote the static BucketOwner answer without resolving the
bucket, so a bucket that does not exist looked like one that does.
Check the bucket first, as PutBucketRequestPayment and the other static
configuration getters already do.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3api: test GetBucketRequestPayment on existing and missing buckets
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* filer: release the stamps of parent directories a rename creates
* filer: test that a suppressed create keeps the moved entry's stamp in flight
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* operation: prepare upload content before assigning the volume
* operation: keep the existing upload function names for prepared payloads
UploadData and doUpload prepare once and hand the prepared payload to
retriedUploadData, so the retry loop, doUploadData, uploadChunk and
uploadChunkToHolders take a preparedUpload directly instead of growing
a prepared twin of each and branching on nil.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* operation: send ciphertext whenever the prepared payload carries a cipher key
doUploadData chose the cipher branch from the option, so a payload
prepared under a different option could be sent as plaintext metadata
or lose its key. Decide from the prepared payload itself.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* operation: build a chunk's preparation and upload options in one place
The chunk loop prepared with a hand-written option while uploadChunk and
uploadChunkToHolders built their own, so a setting that shapes the bytes
could reach one and not the other. chunkUploadOption now feeds all three.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* operation: test that a reassigned chunk reuses its prepared payload
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
The Go volume server fails the RPC when the remote read returns a
different length than req.Size; the Rust server cached whatever came
back as a full-size chunk.
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* wdclient: let master-pushed adds replace the gateway's lookup-fed locations
After a master leader change the S3 gateway clears its FilerClient cache and
refills it through filer lookups. The filer's own master client may still
answer from the previous leader's generation until the new leader has heard
from the volume servers, so a volume server lost together with the old leader
is cached again. The new leader never knew that server and never pushes its
removal, and the gateway keeps trying it until reads fail on it volume by
volume, one 30 s dial timeout at a time.
The gateway's master client relearns every volume from the new leader's
adds. On each add, cache the volume where the master client now knows it to
be, in place of what a lookup cached, and keep lookups already in flight from
caching their answer over it. A provider answer for a volume is now cached
whole or not at all.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* wdclient: take a lookup's fill epoch before its second cache check
A master add that landed between the second cache check and the read of
the fill epoch was invisible to both: the check had already missed, and
the epoch the lookup then read already included the confirmation, so a
filer answer that still named the lost server was merged back into the
confirmed entry. Taking the epoch first means such an add either shows
up in the check or stops the answer from being cached. The same order
covers a reset or clear in that gap.
TestConfirmDuringLookupIsNotUndone fires a confirmation from a test hook
at that point and fails with the old order.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* wdclient: gate the master lookup provider's cache writes on the map epoch
The provider wrote each answer straight into the vid map, so a lookup sent
before a master reset and answered after it landed in the new generation,
which a gateway's FilerClient now reads as the current master's view. The
answer now goes through fillLocations, which drops it once a reset passed.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* wdclient: keep the vidMapClient doc comment on its type
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
* s3: answer ClientDisconnected for a GET the client aborted before the first byte
Fixes#12706
* s3: tell a gone client on TLS and HTTP/2 too, and not on a connection the server closed
* filer: fail health check when absent from lock ring
If the master broadcasts a lock ring that does not include this filer
(e.g. the late-cleanup race in seaweedfs#11442), every lock acquisition
fails with "no lock server found" and locked writes, including S3 PUTs,
hang indefinitely while /healthz and /readyz still report OK.
Make the filer health handler return 503 once this filer has been
missing from its own lock ring for longer than a 30s grace period, so
Kubernetes liveness probes restart it and it re-registers with the
master. The grace period covers startup before the first ring update
and the master's ring-change batching delay.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* filer: hold the lock-ring health clock until the filer announces
AnnounceOnceCaughtUp can hold the master registration for up to a minute
while HTTP already serves, so the filer cannot be in the ring yet. Report
healthy until the announce gate opens and start the grace period after it.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* filer: keep a probe racing a ring rejoin from reading a reset clock
A probe that loses the CompareAndSwap to a concurrent healthy probe could
load the reset zero and report the filer absent since the epoch.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* filer: trim the lock-ring health comments
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* filer: restart the lock-ring health grace when a ring update brings the filer back
The clock only reset when a probe saw the filer in the ring, so a rejoin
and a second removal between two probes reused the first absence and
failed the next probe early. Reset it from the ring snapshot callback.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* filer: test the announce gate leaves the lock-ring health clock unset
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* filer: re-read lock-ring membership after starting the health clock
A probe whose first read raced a rejoin could start the clock after the
snapshot callback reset it, leaving a stale start for the next removal.
The ring is installed before the callback runs, so a second read sees the
rejoin whenever the reset came first.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* filer: treat a health clock changed during the probe as a rejoin
A rejoin and a second removal can both land between the probe's two
membership reads. The second read then shows the filer absent again, and
the probe judged the new absence by the expired start it had read before
the rejoin's reset. The snapshot callback resets the clock before the next
ring update is installed, so a clock that no longer matches the probe's
start means a rejoin happened; report healthy and let the next probe use
the fresh clock.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* filer: test ring updates that land during a lock-ring health probe
Move the absence clock into lockRingAbsence, which takes the membership
read as a function, so the test can install ring updates between the
probe's reads: a rejoin after the first read, and a rejoin plus removal
between the two reads.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* filer: give every lock-ring health reset a new clock value
A probe that saw the filer absent started the clock with a CAS from 0,
so a rejoin that reset the clock to 0 again before that CAS was
invisible, and the probe could install a start time from before the
rejoin. Each reset now stores a new non-positive value and the probe
swaps from the value it loaded before the membership read, so any
rejoin during the probe makes the CAS fail. This drops the second
membership read and never waits on the LockRing mutex.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* filer: test lock-ring rejoins before and after the probe's membership read
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* filer: keep the lock-ring version lookup out of the unhealthy health path
LockRing.Version takes the ring mutex, which SetSnapshot holds while
the snapshot callback transfers locks, so an expired-absence probe
could wait on those transfers and time out instead of answering 503.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* filer: test an expired-absence health probe while the ring mutex is held
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
* test(fuse): two mounts staging the same files without -dlm
Two jobs that extract the same archive into one shared directory, each
through its own mount and without -dlm, both create many of the same
files: a file that exists is skipped, otherwise it is created, written
and closed, then chmod'ed and given its mtime. Before #12593, #12594 and
#12596 the attribute update and the content flush sent the mount's whole
cached entry. Sent after the other mount had rewritten the file, the
update put back chunks that the rewrite had dropped and the filer had
deleted, so the entry pointed at deleted needles and the file failed to
read once its volume was vacuumed. Nothing told the writers.
TestStageSameFilesWithoutDLM runs that on the fuse_dlm cluster with the
mounts started without -dlm (startTestCluster takes a dlm flag;
startDLMTestCluster is unchanged) and checks every staged file: each
chunk is still on the volume server (HEAD on the fid, without
readDeleted) and the filer serves the bytes written. What the losing
writer is told depends on the version and is only logged.
With a 4.53 weed binary the test fails: 66, 71 and 70 of the 400 files
were broken in three runs. With master it passes in about 48 s.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* test(fuse): bound the staging test's filer and volume checks
The checks used the default HTTP client and a background context, so a
server that accepts a request and stops answering held the test until the
package timeout instead of failing the check.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* test(fuse): fail the staging test unless both mounts wrote the same files
A mount whose MkdirAll failed returned early, and a mount that started
late could skip every file, so the test passed with one writer. Start both
loops together, fail on setup errors, and require files both mounts wrote.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* test(fuse): bound the staging test's checks with one deadline
A per-call timeout still let 400 hung lookups and reads run for hours.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
* s3: keep serving for -preStopSeconds after SIGTERM
On SIGTERM the gateway closed its listeners at once, so requests a load
balancer or Kubernetes Service still routed to it failed until the endpoint
was removed. With -preStopSeconds set, the gateway makes /status, /healthz
and /readyz return 503, keeps serving for that many seconds, then runs the
existing graceful shutdown. 0 (the default) keeps the old behavior.
The flag is registered for weed s3 and as -s3.preStopSeconds for weed
filer, weed server and weed mini. A cancelled weed mini cluster context
ends the wait early. The Helm chart passes s3.preStopSeconds and
filer.s3.preStopSeconds through.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3: assert the preStopSeconds chart render under set -e, drop timing from the probe test
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3: wait before weed mini cancels its clients; size grace periods in the chart
weed mini's interrupt hook runs before the S3 one and cancels the S3
gateway's shutdownCtx, so the pre-stop wait never happened there. The wait
is now also registered with beforeMiniClientsShutdown, which runs before
that cancel, and a sync.OnceFunc keeps the interrupt hook from waiting a
second time. Outside weed mini the registration is a no-op.
Helm: the s3 deployment fails to render when terminationGracePeriodSeconds
is below preStopSeconds + 15. The filer statefulset and the all-in-one
deployment, whose grace period is fixed at 60s, add preStopSeconds to it.
allInOne.s3.preStopSeconds is passed through, inheriting s3.preStopSeconds.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3: keep 64-bit counters aligned on 386; let allInOne.s3.preStopSeconds=0 disable the wait
draining sat ahead of inFlightDataSize and inFlightUploads in S3ApiServer
and shifted them off 8-byte alignment on 32-bit builds, so the atomic
upload-limit counters panicked there. It now follows them.
Helm: allInOne.s3.preStopSeconds inherits s3.preStopSeconds only when null;
default treated an explicit 0 as unset. The s3 values and the grace period
error now say a lifecycle preStop hook's time comes on top.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* grace: add OnInterruptFirst for hooks that must run before any shutdown
Interrupt hooks run in reverse registration order, so a hook cannot run
ahead of a service that registers later, such as a volume server that is
still loading its volumes.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3: run the -preStopSeconds wait before every other interrupt hook
Under weed server the volume server registers its hook once its volumes
are loaded, often after the S3 gateway. Its hook then ran first and shut
the volume server down, so the gateway spent the whole wait serving
requests that could not read or write objects.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
* s3: escape raw semicolons before the router matches a route
registerRouter applied EscapeSemicolonsInQuery with router.Use, but mux
runs router middleware only after it has picked the route. Queries()
matchers split pairs on ';', so GET /bucket?uploads;x=1 reached the
multipart listing handler while the handler and the IAM resolver read
uploads%3Bx=1 and saw no uploads subresource. Wrap the S3 router with the
escape so routing and authorization read the same query.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3api: report a listing without prefix as s3:prefix "" in policy conditions
The advanced IAM path already sets s3:prefix to "" when a bucket listing
names no prefix, as AWS does, but bucket policies and attached policies
read conditions from ExtractConditionValuesFromRequest, which leaves the
key out. The AWS home-directory policy that grants s3:prefix ["", "home/"]
then denied the root listing, and a Deny keyed on the empty prefix never
applied.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3: test the served handler composition for the semicolon escape
Move the escape and the Vectors split into s3Handler so a command test can
send ';' queries through the handler the server actually serves.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3: keep the isVectorsRequest doc comment on its function
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* fix(s3api): complete bucket-listing IAM condition handling
Populate s3:delimiter and s3:max-keys for bucket. Add regression tests for listing conditions and variants.
Assisted-by: gpt-6.1-sol, deepseek-v4.1-flash
* fix(s3api): ignore max-keys for multipart listing policies
Assisted-by: GPT-6.1 Terra
* s3api: drop max-keys from multipart-upload listing policy conditions
Bucket policies and embedded IAM policies read condition values through
ExtractConditionValuesFromRequest, which still exposed max-keys on ?uploads
listings. Those page by max-uploads, so an unused max-keys parameter could
satisfy an s3:max-keys condition. Match the advanced IAM path and leave the
key absent there.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* topology: never report negative free volume slots
Free volume slots are computed as max + remote - volume - ecSlots. When phantom or over-registered volumes push the count past the configured max, the result is negative and rolls up into the cluster total, causing volume grow and assign to fail with a negative 'volumes left' count even while healthy servers have capacity. Clamp every free-volume count at zero so an over-committed server contributes no slots instead of subtracting from the cluster.
* topology: drop the issue reference from the free-slot test comment
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* topology: test free slots next to an over-committed server
A server with more volumes than slots must not hide the free slots of
the other servers in its rack, data center or topology.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* topology: sum free slots per data node in racks, data centers and topology
Clamping the rolled-up counters only clamped their sum, so one server with
more volumes than slots still hid the free slots of every other server in
its rack, data center and topology. Interior nodes now add up their
children's free slots, so each data node counts as zero at worst. A data
node in maintenance mode no longer adds slots to its parents either.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* topology: test that disk info keeps an over-committed deficit
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* topology: keep the signed free volume count in disk info
EC encode retries and the balance planners add refunds and planned moves
to the published FreeVolumeCount. Clamping it at zero dropped an
over-committed disk's deficit, so a refund could make a full disk look
free. Volume growth already reads the clamped AvailableSpaceFor.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* topology: test placement next to an over-committed server
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* topology: accept either data center order in the placement spread check
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* seaweed-volume: expect a fresh redb file to start at 72 KiB
The assertion fails on redb 4.2.0, whose empty database is about 1 MiB.
* seaweed-volume: bulk-load redb rebuilds with the stable cursor
redb 4.6.0 has no experimental_cursor feature. Full rebuilds insert
sorted needle ids through insert_before, and a new .rdb starts at 72 KiB.
* seaweed-volume: test which redb files are still at the old floor
The gate is missing, so the test does not compile yet.
* seaweed-volume: decide when a reused redb is still at the old floor
Compact only a file above 72 KiB and at most 1 MiB plus 8 KiB, and only
when floor_compact has not been recorded.
* seaweed-volume: test one-shot compact of an old-floor redb
Reuse, shrink, and storage-error recovery are specified. The helpers
they call are not implemented, so the tests do not compile yet.
* seaweed-volume: compact a reused redb that is still at the old floor
One compact runs when the file is above 72 KiB and at most 1 MiB plus
8 KiB. A storage error drops the handle and opens the same path again
so the volume does not keep a PreviousIo latch.
* seaweed-volume: publish a redb replay before compacting the old floor
compact() is what would make a Durability::None tail durable, and a
storage error then drops that tail on reopen. An in-band file is
committed first. The floor_compact flag is read only inside the band,
so a meta error cannot rebuild a larger index.
* seaweed-volume: drop the redb experimental cursor CI step
redb 4.6 has no redb-experimental-cursor feature. The needle-map tests
that used it now run in the normal cargo test step.
* seaweed-volume: drop the experimental-api-5 notes on redb reads
redb's experimental-api-5 can no longer be enabled, so these reads always
use the inherent ReadOnlyTable::get.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* seaweed-volume: drop the always-false flag from the old-floor band check
Both callers pass false and read floor_compact on their own, so the band
check only needs the file length.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* master: unregister a volume server whose heartbeats stop
A volume server sends a heartbeat every 5 s. When its host hangs or its
network drops packets, its SendHeartbeat stream stays open, and the
master unregisters it only once gRPC keepalive closes the connection,
60 s plus 20 s after the last frame. Until then the master keeps
assigning writes to volumes with a copy on it, and those writes hang.
SendHeartbeat now ends the stream when no heartbeat arrives for three
pulses (15 s), which unregisters the server through the existing
deferred path and tells clients its locations are gone. The server
reconnects and registers again once it can reach the master.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* master: test that clients learn a silent server's volumes are gone
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* master: fail the silent-stream receive test instead of hanging
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
* pb: fall back to TCP when the local gRPC socket cannot be dialed
A temp cleaner can delete a local gRPC socket file while its server keeps
running. Every later dial to that server then failed with "no such file or
directory", so the master could no longer reach its own volume server and
volume growth stopped. Dial the server's TCP address when the socket dial
fails, as the S3 gateway already does for the filer's HTTP socket.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* pb: recreate a local gRPC socket whose file was removed
Once its socket file was deleted, a server kept accepting on an
unreachable listener until restart, so same-host RPCs stayed on the TCP
fallback. Check the path every minute and, when the file is gone, close
the old listener and bind the socket again.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* pb: retry a failed local socket rebind on the next check
A failed rebind ended the watcher, so the socket stayed missing after the
cause cleared. Bind the replacement before closing the old listener: a
failed bind keeps the old listener serving and is retried on the next
check, and the old Serve still reports server shutdown. The old listener
no longer unlinks the path on close, since the replacement now owns it.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* pb: set the local socket unlink policy before serving the listener
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* server: add -grpc.localSocketDir for the local gRPC sockets
`weed server` and `weed mini` hard-coded each local gRPC Unix socket
under /tmp. systemd-tmpfiles ages /tmp after 10 days and macOS after 3,
so a long-running server loses its socket, and RegisterLocalGrpcSocket
keeps routing local dials to the missing path: the master can no longer
reach its volume server.
Add -grpc.localSocketDir to both commands, defaulting to /tmp so an
existing setup is unchanged, and build every socket path from it with
filepath.Join. A folder that does not exist or is not a directory is
refused at startup rather than at the first local RPC. A folder whose
socket paths exceed the platform's sockaddr_un limit is refused too,
because every bind then fails with EINVAL while the server keeps running
and each local RPC dials a socket that does not exist.
Refs #12692
* server: refuse a -grpc.localSocketDir the server cannot bind in
An existing folder the server's user cannot write to passed the startup
check, and then every socket bind failed: ListenOnLocalSocket only logs
the error while GrpcDial keeps routing local dials to a socket that was
never created. After the path-length check, create and remove a probe
Unix socket in the folder, and refuse startup with the bind error if
that fails.
* server: give the -grpc.localSocketDir probe a random name
A probe named only by the process id could be precreated by another
user in a shared /tmp, or left by a crashed process whose pid was
reused, and the server cannot remove such a file, so startup failed
although the real socket paths were free. Name the probe with 8 random
bytes, retry a name that is taken, and rely on the listener's close to
unlink it.
* server: keep the -grpc.localSocketDir probe name within the service socket lengths
The probe was 5 bytes longer than the master and volume socket names, so a
folder whose service sockets fit the sockaddr_un limit could still be refused.
A 23-byte probe name is shorter than any service socket name, and a process
with no local sockets skips the check.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* server: trim the -grpc.localSocketDir comments and drop issue references
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Brent Pease <179639673+brentDSL@users.noreply.github.com>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* webhook: share the default target settings
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* webhook: managed webhook targets with a drainable queue
Targets kept as JSON (name, endpoint, token, filters, retry and worker
settings) build their own queue; Close waits for queued and in-flight
deliveries before stopping it.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* filer: run webhook targets stored in /etc/seaweedfs when notification.webhook.managed is set
The targets in /etc/seaweedfs/notification_webhooks.json start with the
filer and are rebuilt from the store whenever the file changes on any
filer, alongside notification.toml. The previous queue drains before it
stops. Events for the file itself, which holds bearer tokens, are not
sent to notification queues.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* admin: manage filer webhook targets
A Webhooks page under Management lists, adds, edits and deletes the
targets kept in /etc/seaweedfs/notification_webhooks.json. Bearer
tokens are write-only: the API reports whether one is set, and an empty
token on edit keeps the stored one.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* filer: keep the webhook queue when the stored targets did not change
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* filer: keep a replaced webhook queue open until its senders hand over
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* admin: read chunk-backed webhook targets before editing them
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* shell: keep the stored webhook targets out of fs.meta.notify
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* webhook: reject negative managed target settings
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* webhook: no signal handler on queues that reloads replace
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* filer: swap the webhook queue without waiting behind blocked senders
The read lock held across SendMessage made a reload wait for any sender
blocked on a saturated target, and the waiting writer stalled every
other filer write with it, which is when an operator disables that
target. Close now refuses new sends and waits for the senders already
inside SendMessage, and a send that hits the closing queue retries on
its replacement.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* webhook: release blocked test handlers before the server shuts down
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* filer: always run the webhook targets managed in the admin UI
Drop the notification.webhook.managed switch: every filer loads and live
reloads the stored targets. Without stored targets no queue is created.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* scaffold: place the admin-managed webhook note above enabled
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* scaffold: separate the admin-managed webhook note
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* scaffold: note that admin-managed webhooks follow the notification queue
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* filer: retry loading webhook targets after a failed read
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* filer: retire the admin-managed webhook queue on shutdown
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* ci: run the Python 3.9 jobs on 3.10
ubuntu-latest now resolves to Ubuntu 26.04, which has no Python 3.9 build
for actions/setup-python, so the Ceph S3 tests and the PyArrow Parquet
3.9 job fail during setup. 3.10 is the oldest version built for 26.04.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* test: list the Python versions the parquet CI runs
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* admin: apply the saved theme inline at the top of the page head
The theme script was an external file, so a reload had to fetch or
revalidate it before the root got its theme, and the page could paint
with the light default in the meantime.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* admin: set color-scheme on the root together with the theme
Until the stylesheets load, the browser paints the page canvas from the
root's color-scheme, which only the stylesheets set, so a dark theme
showed a white canvas for the duration of every reload.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* ec: mount shards against the index .vif when the local one has no EC config
A worker encode can send shards to a disk that still holds the source
volume's .vif while the encode's .vif lands with the index on a sibling
disk. The shard disk mounted through the source's config-free .vif, so
it reported encode time 0 and fell back to the sidecar layout, and reads
carrying the encode time were rejected as a different encode run.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* volume(rs): mount shards against the index .vif when the local one has no EC config
Mirrors the Go mount: a data-dir .vif that records no EC config yields
to the idx dir's when that one records it, and load_vif_info_across_dirs
reports the directory the .vif actually came from.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* storage: do not mirror EC sidecars onto a disk whose .vif has no EC config
Startup mirroring copied .ecx/.ecj next to a source volume's leftover
config-free .vif and kept that .vif, so after a restart the shard disk
mounted self-contained with encode time 0. Leave such a disk on the
cross-disk mount, which reads the owner's .vif.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* volume(rs): do not mirror EC sidecars onto a disk whose .vif has no EC config
Mirrors the Go startup mirror: a shard disk holding a source volume's
config-free .vif stays on the cross-disk mount, which reads the owner's
.vif, instead of mounting self-contained with encode time 0.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* ec: take the encode .vif from a sibling data dir when the index dir is shared
With a separate -dir.idx the .ecx is shared by every disk, but the encode's
.vif lands in the data dir of the disk that received it. A disk still holding
a source volume's config-free .vif mounted through it at generation 0.
NewEcVolume now accepts the server's data dirs and takes a sibling's EC .vif
when the index is not local and every sibling EC .vif names the same run.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* storage: give EC mounts the data dir of every disk on the server
NewStore records every data dir on each DiskLocation before loading, so the
startup scan, cross-disk reconcile and MountEcShards all pass them to
NewEcVolume.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* volume(rs): take the encode .vif from a sibling data dir when the index dir is shared
Mirrors the Go mount: EcVolume::new_with_vif_dirs takes a sibling's EC .vif
when this disk's own records no EC config, the index is not local, and every
sibling EC .vif names the same run.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* volume(rs): give EC mounts the data dir of every disk on the server
main sets Store.store_dirs before the first add_location, so a disk loaded
before its siblings still sees them; each DiskLocation passes them to
EcVolume::new_with_vif_dirs.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* storage: size startup EC validation by the sibling encode .vif
With a shared -dir.idx, a disk that still holds the source .dat and its
config-free .vif read the default data-shard count, judged custom-ratio
shards an interrupted encode, and removed them along with the shared .ecx
before the mount could take the sibling's .vif. The startup lookup now
falls back to the same sibling .vif the mount uses.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* volume(rs): size startup EC validation by the sibling encode .vif
Mirrors the Go change: validate_ec_volume falls back to the sibling data
dir's encode .vif before judging shards beside a source .dat.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* storage: keep EC shards when sibling .vif files disagree on the encode
SiblingEcConfigVif reports when a sibling .vif cannot be read or the
siblings name different encode runs. The startup data-shard lookup then
returns 0 instead of the default ratio, and validateEcVolume keeps the
files rather than sizing custom-ratio shards by a ratio they do not use.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* volume(rs): keep EC shards when sibling .vif files disagree on the encode
Mirrors the Go change: sibling_ec_config_vif returns Err for an unreadable
sibling .vif or disagreeing runs, ec_data_shards_from_vif returns 0 then,
and validate_ec_volume keeps the files.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* storage: release the opened EC shard when its volume cannot be built
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* volume: keep reads over the download limit local when readMode is local
With -readMode=local the server never sends a read for a volume it does
not hold elsewhere, but a read over -concurrentDownloadLimitMB on a
replicated volume was still redirected to another replica. Skip the
replica fallback in local mode, so the read waits for a download slot
and answers 429 when the wait times out.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* volume(rs): keep reads over the download limit local when readMode is local
Mirror the Go volume server: with readMode local, a read over the
download limit waits for a slot instead of being redirected to a
replica. proxy_or_redirect_to_target no longer needs the local-mode
redirect switch.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* volume: check download capacity under the wait lock
waitForDownloadSlot read the in-flight size outside the cond lock and
the read handler broadcast without it, so a download finishing between
the check and the wait was missed and the read waited out the timeout
for a 429. Check capacity with util.WaitUntil under the lock and
broadcast under it.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* volume(rs): register for the download notification before checking capacity
The wait loop created its Notified future only after the capacity
check, so a body dropped in between had its notify_waiters() lost and
the read waited out the deadline for a 429. Create the future first,
then check, as Go now checks under the cond lock.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* admin: fix cluster submenu toggle
* admin: update embedded JavaScript
* admin: drop the object store submenu handlers
The sidebar no longer renders an object store submenu, so its expand,
link and toggle handlers match nothing; the toggle handler repeated the
manual collapse that fought Bootstrap on the cluster submenu.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: ivan <i.stepanov@navigine.com>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
sudo switches its real uid to root while it starts, so kill -0 from the
invoking user fails with EPERM even though the mount process is alive,
and the wait loop reported "weed mount exited early" a few milliseconds
after starting it.
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* filer: serve caught-up aggregated subscribers from memory, not a held disk pass
An aggregated SubscribeMetadata stream that does not take metadata chunks
(filer.sync, filer.backup, filer.meta.backup, filer.meta.tail, the
filer.remote.* syncs, the S3 gateway's subscriptions, and mounts that
predate chunk support) re-ran the persisted-log pass whenever its cursor
moved: after every memory hold at the delivery watermark and on every
2 s re-probe. The pass holds at the peers' flush low-watermark. Peers
flush on their own one-minute schedules, so on a cluster that keeps
writing some peer has always flushed past a caught-up cursor while the
slowest has not: the pass held at its first entry, the loop came back
to it, and memory was never read. The stream got each change only once
the slowest peer's flush covered it, in bursts up to a flush interval
late, and every pass listed and re-read the newest log files.
When the aggregated ring was already filling at the cursor and evicted
nothing after it, memory holds everything the persisted logs could add,
since every peer's log reaches the ring through the aggregator. Skip the
pass and read memory, held at the delivery watermark as before. The
skipped pass proves nothing past the cursor, and the next-day scan for
an empty pass does not apply. An empty ring, a ring that starts past the
cursor, and a cursor memory just refused still take the pass. Chunk
subscribers already skip it for a caught-up cursor: their ref listing
stops below the flush hold.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* filer: keep a new peer's pre-start history on the disk pass
The ring only receives a peer's events from the start of its subscription.
A peer followed for the first time after this filer started (a filer added
to the cluster, a restarted pod with a new address, or the late-arriving
peers bootstrap) starts at its notice time, so what it logged before then
is only in the persisted logs. A cursor below that start passed the ring
check and skipped the disk pass, losing those events for subscribers that
do not take metadata chunks.
Record the newest start of a peer subscription that had no resume point and
send cursors below it through the disk pass as before. Peers that rejoin with
a resume point (a restart under the same address) do not move it.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
* helm: pin the 4.53 tag in the vector metrics rejection test
The 4.54 release made 4.54 the chart's default image, and that image
serves weed-vector metrics, so rendering vector.metricsPort with the
default tag no longer fails. Pin image.tag=4.53 for the rejection case
and check that the default image renders the metrics port.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* helm: describe vector metrics support from 4.54 on in the chart README
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* helm: say the vector metrics render guard skips image overrides
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* helm: drop the 4.53 vector metrics render guard
The chart now defaults to 4.54, whose weed-vector serves --metrics-port, so
the check keyed on one old tag only blocked valid installs. The test asserts
the default image renders the metrics flag and HTTP probes.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* helm: describe vector metrics without the render guard in the chart README
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* log_buffer: let a subscriber skip wake-ups for unwanted partition keys
Every append woke every registered subscriber. SetSubscriberWakeFilter lets
a reader sleep through appends whose partition key it would filter out; it
still reads them on its next poll. Flushes and keyless appends wake everyone.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* filer_pb: decide subscription matches from an event's directories
MetadataDirectoryMayMatchSubscription rules out events whose directory no
child path of can match the subscription, and ScanMetadataEventDirectories
reads an event's directory and new parent path without allocating.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* filer: wake subscribers of a move's destination directory
Appends are keyed by the source directory, so a filtered subscriber of the
destination would otherwise only see the move on its next poll.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* filer: wake metadata subscribers only for directories they can match
Every metadata event woke and was scanned by every subscriber, so filer CPU
grew with the number of subscribers watching other paths. Readers now
register a directory wake filter, and the per-entry prefilter rules entries
out from their directories before the skeleton scan.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* filer: test that moves into a subscribed prefix pass the prefilter
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* docs: explain the 3-filer recommendation in the production setup notes
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* docs: note that a filer load balancer must forward the gRPC port
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* s3api: create a multipart upload's directory on the object's owner filer
CreateMultipartUpload wrote .uploads/<id> to the gateway's own filer while
the upload's completion commits on the object's write owner. With
independent filer stores, a part or completion reaching another gateway
before peer metadata replication caught up could not see the upload.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3api: look up a multipart upload on the object's owner filer
UploadPart, UploadPartCopy, ListParts, CompleteMultipartUpload,
AbortMultipartUpload and the bucket policy SSE lookup read .uploads/<id>
from the gateway's own filer, so a request on another gateway shortly
after CreateMultipartUpload answered NoSuchUpload until replication
caught up. Read it preferring the object's write owner, like object reads.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3api: write multipart parts to the upload's owner filer
UploadPart and UploadPartCopy stored part entries on the gateway's own
filer, so ListParts and CompleteMultipartUpload on another gateway could
miss a part that replication had not delivered yet. Store them next to the
upload directory on the object's write owner.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3api: list and clean up a multipart upload on its owner filer
ListParts, CompleteMultipartUpload and AbortMultipartUpload listed and
removed part entries on the gateway's own filer. Do it where the upload
and its parts now live, so completion sees every part and cleanup does
not miss entries replication has not delivered yet.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3api: test a multipart upload against a lagging gateway filer
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3api: check the upload's owner before freeing a failed part's chunks
A part write that the owner stored but answered with an error went to
recovery with no object, so the owner was never among the filers checked.
When the owner is outside the gateway's filer list, the others' not-found
answers freed chunks the stored part still references.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3api: list an upload's parts on the prior owner after a ring change
The upload record lookup already falls back to the key's prior owner while
a ring change is in its cooling-off window, but the part listing read only
the new owner. ListParts then reported no parts for an open upload. Fall
back the same way when the new owner has nothing yet.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3api: find the upload record on the prior owner past a bare directory
A part routed to a new owner that has not received the upload yet makes
that filer create .uploads/<id> without the upload's key. The post-write
check then read it as an aborted upload and removed the part. While the
ring change is in its cooling-off window, a record on the prior owner
now wins over such a directory.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3api: merge the prior owner's parts into an upload listing
During a ring change's cooling-off window the new owner may have received
only some of an upload's parts. Listing returned the new owner's parts
alone once it had any, so ListParts dropped the rest and a completion
naming them failed with InvalidPart. Both owners' pages are now merged,
the owner's entry winning a shared name, and cut to the page limit.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3api: fail an upload listing the prior owner could not answer
When the new owner has none of an upload's parts yet and the prior
owner's listing fails, the parts may all be on the prior owner. That
listing read as empty, so CompleteMultipartUpload answered NoSuchUpload
for an upload that still exists; it now returns the error.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3api: fail an upload lookup the prior owner could not answer
During a ring change's cooling-off window an upload the new owner has
not received yet, or holds only as a bare directory, read as NoSuchUpload
when the prior owner's lookup failed. That failure is now returned, so
the client sees a retryable error instead of an upload that is gone.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* admin: support [maintenance.s3_lifecycle] in admin.toml
The admin.toml overlay covers vacuum, volume_balance and erasure_coding but
not s3_lifecycle, so the bucket lifecycle job's settings can only be changed
from the web UI and are lost whenever the admin data directory does not
persist. That is the worst combination for this job type: its
meta_log_retention_days knob decides whether bucket rules ever apply to
objects written before the job was enabled, and silently reverting it to the
default makes the rules no-op without any error.
Wire the four admin-side knobs into the existing overlay:
meta_log_retention_days retention window the meta-log replay can reach
walker_interval_minutes minimum gap between full bucket walks
cluster_deletes_per_second cluster-wide delete budget
cluster_deletes_burst token-bucket burst for that budget
All four are admin config values, not worker values: the engine partition is
computed on the admin side and the delete budget is divided across
execute-capable workers at dispatch time. enabled/retry_limit/
retry_backoff_seconds come for free from the shared section handling.
Tested with TestApplyPluginConfigFromTomlS3Lifecycle, which also pins the
retention=0 case: 0 means unbounded and must survive as an explicit 0 rather
than being treated as unset.
* admin: fix meta_log_retention_days guidance in admin.toml scaffold
Both reviewers caught a real error. `0` is not "unbounded": runShard does
retentionWindow := cfg.RetentionWindow
if retentionWindow <= 0 {
retentionWindow = maxTTL
}
so zero falls back to the longest rule TTL, which keeps every replay rule in
replay and promotes nothing to the walk set. Recommending 0 to force full
scans was wrong, and so was the claim that pre-existing objects are only
reachable through the walk set -- the cold-start recovery walk covers them
regardless of the retention value.
Describe zero as "no retention info supplied" and tell operators to set the
window the filer meta-log actually retains. The code change is unaffected:
it only wires the four keys into the existing overlay.
* admin: tie s3_lifecycle walk-set scans to walker_interval_minutes in admin.toml
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* admin: show the s3_lifecycle retry defaults in admin.toml
The s3_lifecycle descriptor sets no retry limit or backoff, so a failed
run is not retried and the scheduler's 5 s backoff applies; the scaffold
listed the vacuum values instead.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* admin: shorten the s3_lifecycle toml section comment
The retention and walker knobs are read by the worker from the admin
values, not computed on the admin side.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* admin: cover cluster_deletes_burst in the s3_lifecycle toml test
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* admin: check the s3_lifecycle retention zero is a stored int64
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* s3 test: evaluate mutation conditions and serve DeleteEntry in the fake filer
The fake ObjectTransaction now checks each mutation's own condition before
applying it, as the filer does, and reports the failed mutation. ETag clauses
read the stored ETag attribute first, like the filer. DeleteEntry removes the
entry so unrouted delete paths can run against the fake.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3 lifecycle: write the suspended expiration marker before deleting the null version
A suspended-versioning expiration deleted the null version first and then
wrote the delete marker. When the pointer read or the marker write failed in
between, the retry found no null version, resolved as NOT_FOUND or
VERSION_NOT_LATEST, and never wrote the marker, so an older version became
the latest.
The marker now goes first. The latest pointer outranks the null object, so
the key reads as deleted once the marker lands, and a null delete that fails
after it leaves a noncurrent null version rather than exposing an older one.
The null delete is pinned to the entry read before the marker: on the owner
it carries IF_ENTRY_EQUAL on that read, and off the ring the entry is re-read
and compared by identity first. A null object written after the read is a
newer write and is no longer deleted. When the null version is the one that
passed the expiration checks, a missing or changed entry at that read
resolves the request as STALE_IDENTITY before anything is written.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3 lifecycle: retry a failed routed null delete instead of deleting on the default filer
A routed null delete that failed fell through to the unrouted path, which
reads and deletes on the default filer. That filer may not have the owner's
newer null object yet, so the fallback could delete it without the owner-side
IF_ENTRY_EQUAL guard. The failure now returns RETRY_LATER; the marker is
already written, so the key keeps reading as deleted.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3 lifecycle: recheck and delete an unrouted null version under the object write lock
Without a ring owner the identity recheck and the null delete were separate
calls, so a suspended PUT landing between them was deleted. Off the ring
writers take the object write lock, so the recheck and the delete now run
under it.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3 test: strip quotes from the stored ETag in the fake filer
The filer normalizes both sides of an ETag comparison; the fake now strips
quotes from the stored ETag attribute as well as from the candidate.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3 lifecycle: retry the null delete when the object lock lookup fails on a routed key
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3 lifecycle: check the suspended marker against the validated version
A suspended expiration that validated a named version wrote its marker
without comparing the pointer, so a null PUT landing after the checks was
hidden, and deleted too when it landed before the null read. The marker now
compares the latest the way the enabled path does and resolves the request
as VERSION_NOT_LATEST.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3 test: count the marker's object write lock in the unrouted null delete
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3 lifecycle: recheck the attempt deadline after reading the null version
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* s3: accept the governance bypass header case-insensitively
botocore and the AWS CLI send x-amz-bypass-governance-retention: True,
so the exact "true" match ignored the bypass and governance-locked
deletes and retention changes failed with AccessDenied.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3: test governance bypass with SDK-style header values
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* s3api: normalize a nil policy value to no values
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3api: evaluate each policy condition against its own values
The process-wide normalized value cache keyed entries by fmt.Sprint of the
value, so ["a", "b"] and ["a b"] (or [] and [""]) shared one entry and a
condition in one policy could be compared against another policy's values.
Normalizing is cheaper than building that key, so drop the cache.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3api: test that look-alike condition values stay distinct
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* s3 lifecycle: bound each LifecycleDelete attempt with its own deadline
Replay and walker deletes used the job context with no request
deadline, so one stalled unary RPC could hold a shard for the whole
job. Each attempt now runs under DeleteTimeout (default 1m), set after
the limiter wait. A timeout while the job is still live is retried like
a transport error and then halts the shard or walk without advancing;
parent cancellation still stops both the attempt and the backoff.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3 lifecycle: admin setting for the delete RPC timeout
Adds delete_timeout_seconds to the S3 Lifecycle admin form and passes
it to both the replay drain and the walker. 0 keeps the 60s default.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3 lifecycle: describe replay and walker timeouts separately in the help text
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3 lifecycle: fail the pass when a replay delete exhausts its attempts
A transport error or attempt timeout that outlived the retries halted
the shard with a nil error, so the job reported success. processMatches
now returns the error (the cursor still holds at the event), and
runShard treats a drain error as a clean stop only when its own context
ended, since an attempt timeout also wraps DeadlineExceeded.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3 lifecycle: test that a replayed LifecycleDelete resolves after a lost reply
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3 lifecycle: skip the expiration write once the attempt deadline passed
A worker retries a LifecycleDelete whose attempt timed out, while the
first attempt can still be running on the S3 server. An attempt that
stalled before its marker write would then stack a second delete marker
after the retry wrote one. The gRPC deadline reaches the handler context,
so check it right before the expiration writes.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3 lifecycle: recheck the attempt deadline after the suspended null delete
The null-version delete can block on the owner filer past the attempt
deadline; the worker may have retried and written a marker by then.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3 lifecycle: finish the suspended marker once the null version is deleted
Skipping the marker after a successful null delete left an older version
as the latest, and the retry then resolved as a no-op without hiding it.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3: let a routed delete marker commit only while its latest guard holds
createDeleteMarkerIf passes an optional condition, evaluated on the
.versions directory entry under the object's lock on the owner filer,
and reports a refused commit as errLatestVersionChanged.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3 lifecycle: write the expiration marker as a compare-and-set on the latest pointer
A timed-out attempt can still be running on the S3 server while the
worker's retry writes the marker. The marker commit is now conditioned
on the .versions pointer as read just before it and skipped when the
latest is already a delete marker, so whichever attempt commits second
writes nothing.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3 lifecycle: resolve the marker as stale when another version became latest
The marker's compare-and-set only covered changes after its own pointer
read, so a version promoted after the expiration checks passed could be
hidden by the marker. The enabled path now passes the validated version
and a conflict retry that finds a different named latest resolves the
request as VERSION_NOT_LATEST.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3 lifecycle: resolve a null expiration as stale once a version takes the pointer
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3 lifecycle: read the marker's latest version the way getLatestObjectVersion does
A pointer names a version only with its file name; an id alone falls back
to the null object, and the null-is-latest signal makes the null version
current. An expiration that validated a named version no longer hides a
null written by a suspended PUT, and a current null under an id-only pointer
now expires instead of resolving as stale.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3 lifecycle: hold the object write lock for an unrouted expiration marker
Off the ring createDeleteMarkerIf cannot carry the pointer guard, so a PUT
committing between the pointer read and the marker write was hidden. The
read and the write now run under the object write lock that unrouted
writers take.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3 lifecycle: share transient-error retries between walker and replay deletes (#12649)
* s3 lifecycle: retry only transient LifecycleDelete failures
dispatchWithRetry retried every RPC error except parent cancellation, so
an authentication, permission or argument error spent the whole retry
budget before surfacing. Retry only attempt timeouts and the failures
util.IsTransientError recognizes (Unavailable, ResourceExhausted, network
resets); everything else returns on the first attempt. A nil response
is now an error instead of a nil dereference.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3 lifecycle: take a delete-rate token for every LifecycleDelete attempt
processMatches waited on the shared limiter once per match, so the
retries behind it ran outside the cluster-wide deletes/sec budget. The
retry loop now waits on the limiter before each attempt.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3 lifecycle: give walker deletes the replay retry policy
WalkerDispatcher sent each LifecycleDelete once, so one dropped
connection aborted the bucket's daily walk while the same failure on the
replay path was retried. The walker now goes through deleteWithRetry and
shares its transient-error retries, per-attempt limiter wait and nil
response handling.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3 lifecycle: retry a DeadlineExceeded the server sends back
A server or proxy can end the RPC with DeadlineExceeded while the
attempt's own deadline is still running. util.IsTransientError rejects
that status, so the delete halted with retries left. It is a timed-out
attempt like the client-side one, so retry it the same way.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3 lifecycle: say walker deletes get 3 attempts in the timeout help
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* security: refuse to start when per-target server allowlists are not applied
A component that dials every server with one option ignored
<target>_allowed_server_* keys, so a config that only set those
accepted any CA-signed server.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* scaffold: say which components scope server allowlists per target
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* security: also catch per-target server allowlists set through WEED_ env vars
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* security: optionally check the dialed host against gRPC server certificates
verify_server_host under [grpc] or a [grpc.<component>] section switches
the client to certificate-and-host verification.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* scaffold: document verify_server_host
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* s3 lifecycle: add a LifecycleDelete client that fails over between S3 fronts
When the current front returns Unavailable, the client closes it, marks it
failed for the rest of the pass, picks the next front that passes
checkLifecycleReady and resends the same request there. Callers cut off by
that switch closing their connection resend on the new front. When no front
is left, every later call returns one error naming the fronts tried.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3 lifecycle: serve the pass through the failover client
Execute hands the replay drain and the walker one failover client built on
the selected front, reports each switch as job progress, and closes the
current connection when the pass ends.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3 lifecycle: keep failing over until a front serves the resent delete
The resend after a switch went straight to the new front, so a second
lost front ended the call even with a healthy third one, and the walker
has no retry of its own. Resend through the failover loop instead.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3 lifecycle: do not recheck fronts skipped when the pass started
The failover client started with no failed fronts, so losing the selected
front checked the fronts selectS3Front had already skipped again, holding
every delete for another readiness timeout per front.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* s3 lifecycle: walk checkpoints take each entry's resume position
Versions of one object share a Path, so a checkpoint set from Path after
the first version made a resume skip the rest. The ListFunc now gives
each entry the start value that continues after it, and leaves it empty
on all but an object's last version.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3 lifecycle: resume a bucket listing where the walk stopped
The filer listing skipped entries with key <= start, but its order is
not key order: each directory emits its .versions groups before its
files, and a directory's children come before a sibling like dir.txt.
Positions are now one (pass, name) frame per directory level, so a
resume seeds each listing and skips exactly what was finished. A version
group is one unit, and a skipped group still keeps its bare null
version out of the second pass.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3 lifecycle: content hash of a walk view
Hashes every non-disabled action of a view and the versioning of their
buckets, so a saved walk position can be tied to the walk it came from:
another rule set, another walk mode, or a bucket that turned versioned
gives another hash.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3 lifecycle: persist daily walk progress and resume it
A walk that failed or was cut short started over on the next pass,
listing again every bucket it had already finished. WalkBuckets now
keeps the finished buckets and each unfinished bucket's position in a
checkpoint file next to the shard cursors, one per shard set, and the
next walk of the same view and shards resumes from it. The record is
saved every 30s and when the walk stops, and cleared when a walk
completes, so the next walk is full again and sees keys added behind
the position; a clear that fails fails the walk rather than leave a
record that would skip buckets.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3 lifecycle: a stuck resumed walk starts over
A bucket that fails at the same place on every pass kept the walk from
completing, so the buckets it had finished were never walked again and
their due objects waited forever. A resumed walk that fails again
without finishing anything clears its checkpoint, and the next walk is
full. A canceled walk keeps it.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3 lifecycle: do not resume a walk checkpoint left for a week
A checkpoint is cleared only by a later walk of the same shard set, so a
set that stopped walking, such as a one-off per-shard walk, left its
record behind, and a walk of that set much later would skip buckets
finished long before. Records now carry their save time and are not
resumed after seven days; a walk that keeps retrying saves on every
attempt and stays fresh.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3 lifecycle: a failed walk without progress leaves its checkpoint alone
A pass canceled while it read the checkpoint failed the read, then
saved its own empty progress over the record it never loaded, so the
next pass started every bucket again. A failed walk that finished
nothing has nothing new to record and no longer writes; the stuck
resume reset still clears.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3 lifecycle: space out walk checkpoint saves after a failed one
The save time advanced only on success, so once the filer refused a
periodic save every following entry tried again and logged another
warning. The interval now counts from the last attempt.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3 lifecycle: a walk checkpoint save that errors may still be on disk
A save whose reply was lost left onDisk false, so a walk that then
completed skipped the clear and the next walk resumed from the stale
record instead of walking everything.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3 lifecycle: keep a stalled walk checkpoint for a day
A resumed walk that failed again without progress dropped its record
right away, so an outage spanning two attempts restarted the whole
walk. Treat the walk as stuck only once its record has not advanced
for a day.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3 lifecycle: a resumed walk counts its earlier buckets as walked
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3 lifecycle: replay a resumed recovery walk from the walk that began it
A recovery or cold-start pass rewinds replay to runNow - maxTTL. When the
walk resumed a checkpoint, its finished buckets were listed in an earlier
attempt, so an object there that was not yet due then and whose write is
now older than maxTTL was neither expired by the walk nor replayed.
The checkpoint now keeps the start of the walk that began it, WalkBuckets
returns that time (its own start for a fresh walk), and runShard floors
the recovery rewind and the cold-start replay at it. A record without a
start is not resumed.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3 lifecycle: a cold start that resumed a walk replays on the next pass
The pass's shared subscription starts at runNow - maxTTL for a shard
without a cursor, before its walk reports an earlier start, so flooring
the cold-start drain at the walk start could not deliver the older
events. Such a shard now saves its cursor at the walk start minus maxTTL
and returns, like the recovery branch, and the next pass replays the
window from that cursor.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3 lifecycle: test that a cursor below the floor opens the next subscription
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3 lifecycle: test the cold-start replay across two Run passes
A cold start whose walk resumed an earlier attempt sees an older event
only on the next pass. The test runs Run twice against a filer that
serves events past the requested floor, with a later in-window event
that would otherwise move the cursor past the older one.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* mount: fail the write when the swap file chunk cannot be created
#7957 added a nil check after NewSwapFileChunk so a failed swap file
creation returns an error instead of panicking. The result was assigned
to pageChunk, a PageChunk interface, before the check, and an interface
holding a nil *SwapFileChunk is not nil, so the check never fired:
SaveDataAt stored the nil chunk and panicked in WriteDataAt.
Check the *SwapFileChunk before converting it. The new test points the
swap directory below a regular file; on master it panics in
SwapFileChunk.WriteDataAt.
Signed-off-by: anant <170962632+antcybersec@users.noreply.github.com>
* mount: grow the file only after its write is buffered
Write set the new size, times, pending append offset and uncommitted
bytes before AddPage. When AddPage failed (now that a swap file chunk
that cannot be created returns an error), the write was reported as
failed but the larger size stayed on the handle's entry, so the next
flush published it with a hole where the data should be.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* mount: report a partly buffered write as a short write
PageWriter.AddPage fills one chunk at a time, so when a later chunk
cannot be created the earlier chunks already hold their bytes. Return
how much was buffered, and let Write grow the size and times over that
prefix and report it, instead of failing the whole write while the
prefix still reaches the next flush.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* mount: report the bytes a failed copy_file_range already buffered
A streaming copy whose page fails partway returned an error even though
earlier pages, or the start of the failing one, were already buffered and
would reach the next flush with the destination size left short. Count
the buffered bytes, grow the destination over them, and return them as a
short copy.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* mount: free the copy test's destination pages
The buffered memory chunk counts toward the process-wide memory chunk
ceiling, so repeated runs of the test pushed later ones onto the
unusable swap directory.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* mount: count only the bytes a swap chunk stored
A swap chunk whose file write fails stores a prefix or nothing, but
SaveDataAt returned no error and AddPage counted the whole span as
buffered. SaveDataAt now reports the short write, and AddPage passes the
stored count up so a write is never acknowledged past it.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* mount: fail a partly buffered writeback write
Kernel writeback ignores the count in a WRITE reply, so a short write
there drops the unbuffered rest without an error. A write flagged as
page-cache writeback now keeps its buffered prefix but returns the
error, which the kernel reports on fsync.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Signed-off-by: anant <170962632+antcybersec@users.noreply.github.com>
Co-authored-by: anant <170962632+antcybersec@users.noreply.github.com>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* install.sh: offer the vector engine on linux only
Releases build weed-vector for linux amd64/arm64 only, so --component vector
on darwin or windows downloaded a missing asset.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* install.sh: add --enterprise
Installs from seaweedfs/artifactory: weed-enterprise-<os>_<arch>[_large_disk],
weed-volume-enterprise[_large_disk], weed-worker-enterprise and
weed-vector-enterprise. The latest tag skips vfs-* releases, and the Rust
volume server is linux-only there, so --component all skips it elsewhere.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* Revert "install.sh: offer the vector engine on linux only"
Releases after 4.53 publish weed-vector for darwin and windows too
(rust-vector-platforms.yml).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* install.sh: Enterprise vector engine is linux-only
Enterprise releases publish weed-vector-enterprise for linux amd64/arm64;
open-source releases keep their darwin and windows engines.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* test: installer --enterprise asset names, platforms and latest tag
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* install.sh: Enterprise latest is the newest plain version tag
artifactory also releases previews and a dev build, and a prerelease such
as kvcache-filer-native-v0.1.0-preview.1 sits in the release list.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* README: show passing --enterprise through bash -s --
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* install.sh: the open-source weed for windows is a .zip too
Releases publish windows_amd64[_large_disk].zip, so asking for a .tar.gz
404s on every Windows host.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* install.sh: install the windows weed as weed.exe
weed-volume and weed-vector keep their .exe on Windows; weed lost it, so
cmd and PowerShell could not run the installed binary.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* install.sh: suggest weed-volume only when it was installed
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* shell: fix s3.circuitBreaker help examples and error messages
The help examples used -type count, but -type accepts Count and MB (as
its flag description says), so both examples failed with "type not
valid, only 'count' and 'bytes' are allowed", an error that names
neither accepted value. Two other errors pointed at flags that do not
exist: -max and -counts, where the flag is -values.
Use -type Count in the examples, name Count and MB in the type error,
and -values in the value errors. TestCircuitBreakerHelpExamplesRun runs
every help example and fails on master for the two -type count lines.
Signed-off-by: anant <170962632+antcybersec@users.noreply.github.com>
* shell: name the bad action and the -actions flag in s3.circuitBreaker
The unknown-action error pointed at a nonexistent -action flag and echoed
the whole -actions list instead of the action that was rejected.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Signed-off-by: anant <170962632+antcybersec@users.noreply.github.com>
Co-authored-by: anant <170962632+antcybersec@users.noreply.github.com>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* filer: report a SQL listing that fails partway through
ListDirectoryPrefixedEntries never checked rows.Err() after its rows.Next()
loop. When the result set failed partway, for example on a lost
connection, rows.Next() returned false and the listing returned the
entries read so far with a nil error, so callers saw a shorter directory
instead of an error. Check rows.Err() once the loop has run to the end,
as the scan in abstract_sql_store_scan.go already does. A listing the
callback stopped on purpose is left alone: rows.Err() would return a
canceled request context's error there.
This covers every store built on AbstractSqlStore (mysql, mysql2,
postgres, postgres2, sqlite).
Signed-off-by: anant <170962632+antcybersec@users.noreply.github.com>
* filer: wait for the canceled SQL listing to release its rows in the test
A fixed sleep let the callback return before database/sql closed the
rows, so a listing that always consulted rows.Err could still pass.
Wait, bounded, until the connection is back in the pool instead.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Signed-off-by: anant <170962632+antcybersec@users.noreply.github.com>
Co-authored-by: anant <170962632+antcybersec@users.noreply.github.com>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
The test unmarshalled the command's output into both actual and expect,
so it compared the output with itself and the expected JSON in each case
was never checked; it also skipped the first case. Compare every case's
output with its expected config as S3CircuitBreakerConfig messages (the
output includes default fields such as "enabled": false that the
expected JSON leaves out), and restore LoadConfig when the test ends.
Signed-off-by: anant <170962632+antcybersec@users.noreply.github.com>
Co-authored-by: anant <170962632+antcybersec@users.noreply.github.com>
* dailyrun: count entries read from the filer, skipped ones included
A versioned bucket's pre-pass and a resumed walk read entries without
emitting them, so a walk can read for hours with no visited entry.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3 lifecycle: count listed entries as progress work
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
A flush uploads the dirty pages without the handle lock, then commits the
entry under it. A write in between grows the entry's size, but its pages
are not in the committed chunks. Clearing dirtyMetadata then made the next
flush skip the commit, leaving a file with the new size and no chunk for
the write, which reads back as zeros.
Count page writes on the handle and clear dirtyMetadata only when none
landed since the flush started uploading.
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* vacuum: report a request dropped because another vacuum is running
The master runs one vacuum at a time. A request that arrives while
another one is running, the automatic vacuum or an earlier request, is
dropped: Topology.Vacuum logs "Vacuum is already running" and returns.
The VacuumVolume RPC and /vol/vacuum then answered as if the vacuum had
run, so volume.vacuum printed nothing and exited 0 with no volume
compacted. Where the automatic vacuum is busy for hours, an explicit
volume.vacuum -volumeId could do nothing every time, and nothing said so.
Topology.Vacuum now returns ErrVacuumRunning for a dropped request. The
RPC returns it and /vol/vacuum answers 409 with it. volume.vacuum names
the request that failed, because a list of volumes or collections is
sent as several requests and a vacuum that starts in between fails only
the later ones. Requests are still dropped, not queued, and the
automatic vacuum ignores the error as before.
The integration tests that need their /vol/vacuum request to run ask
again while the master answers 409.
Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
* docs: note the 409 answer from /vol/vacuum while another vacuum runs
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
* s3 lifecycle: report work-based job progress during a pass
The daily pass sent progress only while starting, so a healthy scan that
ran for many hours could reach the admin's 24-hour no-progress guard.
Worker heartbeats prove the worker is alive, not that this job moves.
While dailyrun.Run is in progress the handler now sends a progress update
every 5 minutes, and only when the pass's counters (entries visited,
replay events, RPC outcomes including errors) moved since the last
delivered update. An idle tick sends nothing, so a stalled pass still
expires. A failed send is retried on the next tick. The reporter is
stopped and joined before the completion is sent, so no progress can
follow the terminal result.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* admin: test that progress, not heartbeats, keeps an old job active
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3 lifecycle: label progress phase and bucket as the last started
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* s3 lifecycle: add a LifecycleReady RPC that only authorizes the caller
LifecycleReady runs the same checkAdminAuth as LifecycleDelete and reads
no bucket or object metadata, so a lifecycle worker can confirm that an S3
front accepts its internal credentials before it starts a pass.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3 lifecycle: check S3 fronts with LifecycleReady before the pass
Execute used to dial the first registered S3 front and start loading
policies and walking buckets, so a missing or mismatched
jwt.filer_signing.key surfaced only as repeated per-bucket delete
failures. It now checks each front in order with LifecycleReady, using
the credentials LifecycleDelete sends, under a 10s bound that also
follows parent cancellation. The first front that passes serves the
pass; fronts that fail are reported in the job's progress and the log.
When none passes, the job fails before any listing with an error that
names the front and says whether this worker has the key set, without
key or token material. Fronts that predate LifecycleReady are checked
with an empty LifecycleDelete, which they authorize and then reject
before any lookup.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* s3 lifecycle: collect per-pass bucket results and work counters
WalkBuckets kept only the first bucket error and logged the rest. A
RunStats shared by the walker, the walker dispatcher and the replay
drain now records every bucket's walk result (walked, failed, skipped
on cancellation), entry and object visits, replay events, and
LifecycleDelete outcomes under the dispatch_total outcome labels.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3 lifecycle: persist the last fully successful pass
Shard cursors and walk timestamps advance on partial passes as well, so
neither says when lifecycle last covered every policy. Keep that in its
own record next to the cursors, plus a worker gauge for it.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3 lifecycle: report each pass's result in the job record
The job ended with a fixed success summary, or with only the first error,
and malformed lifecycle policies were V(1) logs. The completion now
carries the pass status (complete, partial, failed, canceled), bucket
totals and failures, invalid policies, entry, object and event counts,
LifecycleDelete outcomes, the last phase and bucket, the elapsed time
and the last full pass, on failed passes as well.
Only a complete pass advances the full-pass record; a malformed policy
makes the pass partial.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* admin: keep a failed job's result in its record
Completions with Success=false dropped Result.Summary and
Result.OutputValues, so a handler could not report what a failed run
had done, which buckets failed, or how far it got. Keep them; the error
message still drives the job's message and run record.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* grafana: chart the time since the last full lifecycle pass
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3 lifecycle: a replay halted on an unresolved delete is a partial pass
BLOCKED, RETRY_LATER and exhausted transport retries stop a shard's
replay without an error, leaving its due events for a later pass. Count
those shards and report the pass as partial, so it no longer advances
the full-pass record.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3 lifecycle: a pass with throttled walks is not a full pass
WalkerInterval can skip the walk for walk-only rules (ExpirationDate,
ExpiredDeleteMarker, NewerNoncurrent) with no error. Count those shards
and keep the full-pass record for passes that evaluated every rule; the
job reports full_pass and walks_deferred.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3 lifecycle: keep the saved full-pass time when the save fails
Publishing a timestamp that never reached the filer let the gauge move
back on the next job's reload. Keep the previous value and report the
save error in the job result.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* admin: drop a failed attempt's summary when the retry reports none
Retries reuse the job record, and failed completions now keep their
result summary, so a retry that completes without a Result showed the
failed attempt's summary, and a successful one used it as its message.
Clear the summary when a completion carries no Result, as the output
values already are.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* mount: fence a dropped handle only when it holds unpublished content
dirtyMetadata is also set by chmod, xattr and fallocate, so a failed
metadata-only close refused every later write of the inode.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* mount: fence the inode before its last handle leaves the handle map
A reopen between the map removal and the fence could publish at the
stale kernel EOF.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* mount: fence a failed metadata flush only when the handle holds content
A refused synchronous flush or an abandoned writeback metadata flush of a
handle with only chmod, xattr or fallocate changes fenced the inode, so
later writes were refused until FORGET.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* mount: count a size change on an open handle as unpublished content
A truncate or extend through an open handle changes the entry's size and
chunks, but it set only dirtyMetadata. A failed close then skipped the
final-release fence, so a reopen before FORGET could publish against the
untruncated entry.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* mount: mark copied content dirty before each page of a streaming copy
A copy that failed or was cancelled after earlier pages were accepted left
dirtyContent false, so the final-release fence skipped that content.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* mount: mark copied content dirty only after a page is accepted
A copy whose first page is rejected leaves the destination unchanged and
the kernel EOF where it was, so fencing the inode on release only blocked
later writes until FORGET.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* mount: skip the server-side copy while the destination holds unpublished content
A whole-file server copy clears dirtyContent when it commits, so a page a
cancelled streaming copy left behind would lose its release fence. Fall
back to the streaming copy instead.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* mount: skip the server-side copy while the source holds unpublished content
The filer copies the stored source, so pages still pending on the source
handle would be missing from the destination. The streaming copy reads
them through the dirty pages.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* mount: check the publication fence before a server-side whole-file copy
The filer copy publishes the destination without going through the
flush-time guard, so a fenced writeback inode could still take new
content before FORGET. Fall back to the streaming copy, whose flush is
guarded.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
filer: take the listing snapshot as a claim below later and unfinished stamps
A ListEntries snapshot tells the client that every event at or below it is
already in the listing, so the mount drops those events from a directory
build and fences them with the directory floor. The snapshot was a raw
time.Now(), which an event stamp can tie on a coarse clock: on Windows a
mkdir right after a listing of its parent was stamped with the snapshot's
own timestamp, the mount took it as listed, and the new directory stayed
missing from a parent marked cached. A create stamped before the snapshot
but not yet in the store was dropped the same way.
Take the snapshot through the same in-flight claim the idle heartbeat uses:
it stays below every unfinished stamp, and every later stamp lands above it.
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* mount: guard append gaps over concurrent content merges
* mount: fence a writeback inode when its async publication is abandoned
A drain that gives up on a non-precondition error left the token clear, so a
reopened writeback handle could publish past the stored EOF.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* mount: fence a writeback inode when its async data flush fails
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* mount: fence the inode when its last handle drops uncommitted content
A synchronous release whose final flush fails destroys the handle's dirty
pages, while under writeback the kernel EOF already counts them.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* mount: start the close()/fsync commit deadline after the data upload
Flush, Fsync and the truncate path of SetAttr create the 30 s
metadataFlushTimeout context on entry and pass it to doFlush, so the
clock also runs through FlushData, the upload of the dirty pages. That
upload has its own timeouts and retries. When an assign through a stalled
filer or a slow volume server takes most of the 30 s, CreateEntry then
fails at once with "context deadline exceeded" and close() returns EIO
for data that was already stored. The deadline is meant for the commit
alone (#10102: "a healthy CreateEntry completes in well under a second").
doFlush now takes the commit timeout and starts the deadline once the
data is uploaded. close() still cannot hang on a stuck filer: the commit
keeps its 30 s bound. Release and the flush before a rename pass 0, as
before (no overall bound, only the per-call filer timeouts).
Seen in production on a training cluster: a 30 s stall of one replica's
disk, and a filer that stalled for about 30 s, each failed close() with
EIO although the upload had succeeded.
Tests: close(), fsync() and a truncate below dirty data, each after an
assign slower than the deadline (all three fail before this change), and
a filer that never answers CreateEntry still fails close() at the
deadline.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* mount: charge the flush commit deadline only for its filer calls
Starting the 30 s deadline after the data upload is not enough.
flushMetadataToFiler does more than call the filer before it commits: it
waits for the handle's locks, reads the file back and uploads it again
when its chunks are bloated (maybeMergeChunks), and uploads a manifest
once enough chunks have built up (MaybeManifestize). None of that takes
the context. When it outlasts the deadline, CreateEntry again fails at
once with "context deadline exceeded", and close() or fsync() returns an
error for data that was stored.
The deadline becomes a budget, commitBudget, charged only while a filer
call of the commit runs: CreateEntry, and the read of a changed hard
link before it. The attempts of one flush share it, and a flush is not
retried once it is spent, so a stuck filer still fails close() after
30 s. Release, the flush before a rename and the async flush pass no
budget, as before.
Seen in production: the merge of a 1.8 MiB sqlite file with 183 chunks
took 32 s because one chunk's read first tried a volume server that was
down, and the commit failed 32 microseconds after the merge finished.
Tests: a flush whose manifest upload is slower than the deadline commits
(it fails without this change), and the filer calls of one flush share
the budget.
Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
* mount: give the stalled-lookup refusal test the new doFlush argument
TestContentFlushRefusalWithStalledLookup came in from master after this
branch changed doFlush's signature, so the merged package did not build.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
The read of the stored entry ran in the create's filer callback, so a read
that hit the call deadline turned the refusal into a filer timeout, and
both retry layers resent the refused create. Read it in its own call,
bounded to one filer call.
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* util/http: stop a full-chunk read at the chunk's recorded size
weed mount shrinks a chunk's Size in place on ftruncate but leaves its
needle untouched. A view covering the whole shortened chunk is fetched
without a Range header, so the reader returned every stored byte. The
filer GET then wrote past the declared Content-Length and sent an empty
body, and the stream reader served the stale tail.
ReadUrlAsStream and the cipher path now deliver at most size bytes;
size 0 still reads the whole needle for manifest and log chunks.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3api: limit a plain chunk read to the requested size
A full-chunk read sends no Range header, so a chunk shortened by a mount
truncate streamed its whole needle into the S3 response.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* util/http: still report a failure past the chunk size
The volume server aborts a full-needle response when the needle fails
its checksum, and only at the end. Stopping at the chunk size missed
that, so a damaged copy read as healthy. Read the rest of the body and
return its error so the caller tries another copy.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* s3vectors: call the registered weed-vector processes servers
The gateway's discovery named them workers, the name of the admin
plugin workers. Also drops two comments left without a function.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* test/s3vectors: build the Enterprise engine as weed-vector
In an Enterprise checkout, make ./weed-vector builds the distributed
engine, which also serves VectorService, instead of a separate
weed-vector-worker. An open-source checkout builds weed-vector-server
as before.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* test/s3vectors: distributed mode starts two more weed-vector engines
It started weed-vector-worker, which no target builds now, and waited
for a gateway log line that no longer exists. The Enterprise
weed-vector's engines all join the placement controller.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* mount: record a hard link's looked-up names on its inode
Lookup mapped a second name of a hard-linked file to the existing inode
without adding it to the inode's names, so GetAllPaths missed it and the
final forget left it mapped to a released inode.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* mount: keep an open file publishing when a peer unlinks one of its names
A delete event from another client marked the inode's open handle deleted
even when other names still held the hard-linked file, so every later
fsync reported success without saving metadata. The delete event's entry
carries the link count the filer decided on, so an alias delete now drops
only that name, keeps the handle on a name this mount knows, and has the
next flush read the survivor's stored entry before publishing.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* mount: test peer unlinks of a hard link name held open
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* mount: leave an unlinked file out of the periodic metadata flush
flushFileMetadata did not check the handle's delete mark. A file
unlinked while open before its deferred create reached the filer was
created again by the next periodic flush, which expects no entry.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* mount: read a pending link change before a periodic metadata flush
Only flushMetadataToFiler applied linkRefresh. After a peer unlinked one
name, the periodic flush kept publishing over the old link count, so the
filer refused every one until a regular flush ran.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* mount: test the periodic flush after a peer unlinks a hard link name
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
On Windows Lance commits local tables by rename and leaves the manifest
ETag empty, so a table dropped and written again at its location reused
the old table's lease for the same version number.
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
A peer's chunk append now merges into the flush, so these tests no longer
saw the refusal they exercise. A replaced file is still refused.
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* s3: re-check a versioned DeleteObject If-Match under the object lock
A versioned or suspended DeleteObject without a versionId routed to the
owner filer after checking If-Match outside any lock, so concurrent
conditional deletes could all pass the check and each add a delete
marker. Keep conditional deletes on the lock path, which re-checks the
precondition under the object write lock, as conditional versioned PUTs
already do.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3: guard a conditional version delete on the owner filer
DeleteObject with a versionId checked If-Match before routing the
delete to the owner filer, so concurrent conditional deletes of one
version could all report success. Carry the If-Match clause into the
owner's transaction as a guard on the version (the null entry for
versionId=null), evaluated under the object's path lock before anything
applies, and map that guard failing to 412 instead of the object-lock
403. An If-Match the filer cannot evaluate stays on the lock path.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3: carry an If-Match ETag list into the routed delete
A DeleteObject whose If-Match listed several ETags fell back to the
gateway's object write lock, while a single-ETag delete of the same key
ran under the owner filer's path lock. The two locks do not exclude each
other, so a list delete and a single-ETag delete could both report
success. Send every candidate to the filer, whose strong comparison
matches checkDeleteIfMatch, so all conditional deletes with an owner are
decided under the owner's lock.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3: fail a conditional null delete on an owner transport error
The owner may have committed the delete before the error, and the gateway
lock path it fell back to does not exclude routed deletes, so a second
If-Match delete could pass there too. Answer 500 as the specific-version
path does; unconditional deletes still fall back.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3: fail a conditional null delete on any owner error but the nested refusal
Only the owner's non-empty-folder refusal is known to apply nothing. After
any other owner error the delete may have been partly applied, and a lock-path
re-check does not exclude a concurrent routed delete of the same version, so a
conditional null-version delete answers 500 instead of falling back.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3: strip a nested conditional null delete on the owner
A conditional versionId=null delete that the owner refuses because other
keys are nested under the entry used to fall back to the gateway lock and
re-check If-Match there. That lock does not exclude routed deletes: once
the last nested key is removed, a concurrent routed delete can pass the
same check on the owner, and the fallback then treats the missing entry as
deleted, so both requests answer 204.
Demote the entry on the owner instead, in a transaction guarded by the
If-Match clauses and by the entry it was read as, with the object-lock
guards on the request. A missing or already demoted entry answers 412.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* filer: add an IF_OBJECT_EXISTS write condition
IF_EXISTS also holds for a plain directory, which is what is left of a key
once the object on it is stripped while other keys stay nested under it.
IF_OBJECT_EXISTS holds only for an entry that carries an object.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3: require an object for a conditional null delete with If-Match: *
Stripping the null version off an entry other keys are nested under leaves a
plain directory. Once the last nested key is gone, a routed delete with
If-Match: * still found an entry there and removed the empty directory, so a
second conditional delete could also answer 204. The owner now checks that
the entry carries an object.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3: settle a wildcard null delete on the entry when the filer predates IF_OBJECT_EXISTS
A filer older than IF_OBJECT_EXISTS fails it closed, so with the gateway
upgraded first every If-Match: * delete of a null version answered 412. When
the narrowed guard fails, the gateway now reads the entry on the owner: absent
or a plain directory is a 412, otherwise the file is deleted, or the directory
stripped, under IF_EXISTS and IF_ENTRY_EQUAL on the entry it read. The demote
keeps the caller's If-Match clauses, since the entry guard already requires an
object.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* filer: cancel a chunk download that no reader waits for
* filer: stop waiting for reader cache budget once cancelled
* filer: keep a cancelled download's re-lookups off its context
The refresh and retry lookups took the download's context. FilerClient
coalesces concurrent lookups of a volume, so a cancelled download leading
one failed the other readers waiting on it. Fetches still stop on cancel.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* filer: test that cancelling a download leaves its re-lookup running
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* filer: cancel unheld downloads when the reader cache is destroyed
ReaderCache.destroy waited for every downloader, so closing a reader
waited for a stalled prefetch nobody read. Downloads with no reader or
stream on them are now cancelled first; ones still in use keep running.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* filer: test that destroying the reader cache cancels an unread prefetch
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* filer: cancel a destroyed download once its last holder leaves
destroy skipped downloads a reader or stream still held, and once they
were out of the map nothing cancelled them, so destroy waited for a fetch
nobody used. removeConsumedLocked now cancels any consumable download,
whether or not it is still in the map.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* filer: test a destroyed download cancelled after its last stream leaves
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* filer: release the stream only after destroy dropped the download
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* filer: test a cancelled stream leaving a waiting reader's download running
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* s3tables: re-anchor legacy owners for federated STS sessions
A web-identity role session was keyed on its bare OIDC subject until
owners became issuer-scoped. Its sessions now carry the scoped aws:userid
but no asserted issuers, so the legacy subject check never applied and the
session lost the table buckets and namespaces it had created. Read the
configured issuer set for such sessions.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3api: keep token claims from asserting trusted issuers on STS sessions
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* iceberg: give the catalog's table handler the local accounts and issuer set
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3tables: test a federated session on a bucket its legacy subject owns
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* sts: drop a token-supplied aws:userid when a session has no parent user
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3api: take a session's aws:userid from its signed parent user
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3api: test that session claims keep only the server-asserted aws:userid
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3tables: match legacy subject patterns in table bucket policy principals
A policy written before issuer scoping can name role sessions by a pattern
over their bare OIDC subjects. Match such a pattern against the caller's
subject claim once that subject resolves to the session's scoped principal
under the single configured issuer.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3tables: test a federated session against a legacy subject pattern principal
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* s3api: refuse a PUT to a stored version's physical path
In a Suspended bucket a PUT to <key>.versions/v_<id> replaced the stored
version's body in place. Look the entry up when the key has the version
file shape and deny the write when it is a version file.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3api: refuse copy, multipart, POST and rename writes to a version file
Apply the same version file check to the CopyObject, CreateMultipartUpload,
CompleteMultipartUpload, POST Object and RenameObject destinations.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3api: read a version file write target on its logical owner
The guard resolved the key as an object, which on a ring reads the
physical path's owner and, in a versioned bucket, may return a nested
version under <key>.versions/v_<id>.versions instead of the stored
version. Look up the physical entry directly, routed by the logical key.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3api: confirm a version file miss on a skipped owner
When the owner was recently unreachable the routed lookup answers from a
replica, whose miss does not show the version is absent. Ask the owner
directly in that case and refuse the write when it cannot answer.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3api: confirm every routed version file miss on the owner
Whether the routed lookup skipped the owner can change between the lookup
and a second unreachable check, letting a replica's miss through. Ask the
owner after any miss instead.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* fix(mount): retain refused async metadata before releasing handle
* mount: sync the failed-metadata directories with util.FsyncDir
Directory fsync fails on Windows, where the mount also builds, and returns
EINVAL on some filesystems. Either kept the retention loop retrying a record
that could never be saved, so the drain never finished.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* mount: keep failed-metadata records in a per-user directory
The write cache root defaults to the shared temp directory. The first mount
to save a record created failed-metadata there as 0700, so another user's
mount could never save one and its drain retried forever.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* mount: drop the async conflict probe test
TestContentAsyncConflictKeepsRecoveryRecord drives the same refused drain
with the retry backoff stubbed out; the probe slept through the real backoff
(7 s) and its kept-handle branch could not run. Its peer-entry check moves
into the recovery record test.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* filer/etcd: client keepalive and per-request deadlines
An etcd member that goes silent with its connections open (a dead or hung
node, a network partition, a frozen process) keeps the store's requests
balanced onto it waiting forever, because the client has no keepalive and
the calls have no deadline. Writes through the filer stop although the
other members keep quorum.
keepalive_timeout (default 3s, "0s" turns keepalive off) enables gRPC
keepalive with the ping interval fixed at gRPC's 10s minimum. gRPC also
sets TCP_USER_TIMEOUT to it on Linux, so a connection whose sent data goes
unacknowledged that long is closed by the kernel. request_timeout (default
3s) bounds every Get, Put and Delete unless the caller set a deadline;
timeout keeps its meaning as the dial timeout. DeleteFolderChildren gets no
request deadline: its DeleteRange grows with the directory, and keepalive
still ends it if the member goes silent.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* Update weed/filer/etcd/etcd_store.go
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
* Update weed/filer/etcd/etcd_store_test.go
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
* Update weed/command/scaffold/filer.toml
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
* filer/etcd: no request deadline on entry writes
A Put whose client deadline expires may still have been committed. The
filer's upload handler treats the error as a failed write and deletes the
uploaded chunks, which the saved entry may then point at. InsertEntry and
UpdateEntry therefore keep the caller's context: etcd's own request
timeout ends a Put a live member cannot commit, and keepalive ends one on
a member that goes silent. Reads and deletes keep request_timeout; a
timed-out store delete returns before the filer deletes any chunks.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* filer/etcd: no request deadline on KvPut
The filer wrapper stores a hard-linked file's attributes and chunk list
through KvPut. An HTTP append to a hard-linked file whose KvPut times out
after etcd saved it returns an error, and the upload handler deletes the
appended chunks the saved hard link now points at. KvPut keeps the
caller's context like InsertEntry and UpdateEntry.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
When a mount is rooted at an existing directory, two code paths replace
that directory's filer entry with one the mount builds itself:
- every mount start re-creates the root with a non-exclusive mkdir,
which the filer handles as an update of the whole entry;
- creating or removing an entry directly in the root saves the root's
entry to touch its mtime, and maybeLoadEntry synthesizes that entry
from the mount options instead of reading the filer.
The filer keeps only crtime and inode on such an update, so the root
loses its mode, owner and extended attributes. When the root is a
bucket it also loses the bucket quota, which silently drops to 0
(unlimited), and the settings S3 keeps on the bucket entry. The explicit
allowEmptyFolders=false that #11370 meant to keep is replaced by true at
every mount start, because the mkdir has wiped it before
ensureBucketAllowEmptyFolders looks.
Create the root only when a lookup says it is missing, and then
exclusively. Leave the root out of the filer-side mtime touch: its
attributes come from the mount options (setRootAttr), so the filer copy
of its mtime was never visible through the mount anyway.
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* mount: keep flushing an open file whose other hard link survives an unlink
Unlink marks the inode's open handle deleted so no flush recreates the
removed name. When the entry was one of several hard links the inode lives
on under another name, yet the mark stayed and every later fsync and close
returned OK without publishing metadata. Lift the unlink's own mark once
the name is gone and the inode still has a path.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* mount: carry a link's stored entry into the file's open handle
Link rewrites the source entry with the new hard link id, count and times,
but an open handle kept its old base and entry. Its next conditional
content flush was refused (EIO), and an unconditional one would have
dropped the hard link id. Move the handle onto the stored entry when only
the link fields and times changed.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* mount: carry an alias unlink's stored entry into the open handle
Removing one of several hard links lowers the stored link count and
stamps a new ctime, and the handle may still hold the removed name. Look up
the entry at the name the handle flushes through and adopt it, so the next
content flush matches the stored entry, writes the right count, and
attribute saves address the surviving name.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* mount: test an open handle across link and alias unlink
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* mount: read an alias unlink's lost entry again at the next flush
When the read of the surviving entry right after an alias unlink fails,
the handle kept a base with the old link count and ctime, and every later
flush was refused even after the filer recovered. Leave the hard link id
on the handle and have the next flush read the stored entry and adopt it
before publishing.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* mount: publish a queued close before unlinking a name another link keeps
With writeback caching, close only queues the flush. Unlink marked the
handle deleted before waiting for that flush, so the flush skipped its
metadata and the surviving link kept the content from before the close.
When another link keeps the file, wait for the queued flush first.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* mount: let a queued close reach the filer before linking the file
With writeback caching a new file is created on the filer by its queued
close, and a listed directory serves the source from the cache, so a link
right after close updated a name the filer did not have yet and failed
with EIO.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* mount: skip a link change no newer than the handle's entry
A link change carried into the handle could replace a base already moved
past it by a later acknowledgment. Use the same clock-domain check as
installAckedEntry.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* mount: keep a link change's ctime in the dirty entry
The flush does not restamp times, so a dirty entry carrying an earlier
write's ctime moved the filer's ctime back from the link or unlink.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* mount: flush through the file's current name when the resolved one is gone
doFlush resolves the name before uploading the data. An unlink of that
alias, or a rename, in the meantime sent the publish to a name the file no
longer has.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* mount: keep the pending link read when its name is already gone
A flush could read the removed alias name, find nothing, and drop the
pending read of the surviving link, leaving the stale base in place.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* mount: test link and alias unlink across queued closes, flush races, versions and ctime
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* mount: pick a flush's name again once its locks are held
An unlink or rename that lands between the name pick and the locks would
otherwise send the flush to the removed name, where the create guard fails.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* mount: test a flush waiting on its lock through an alias unlink
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* mount: check the flush target after every lock acquisition
A flush whose name kept changing published to the name chosen before its
last lock wait. Check after each acquisition and return a retryable error
once the attempts run out.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* mount: test a flush whose name changes during every lock wait
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* shell: return flag parse errors from every command
48 commands returned nil when their flags failed to parse, so a mistyped
flag printed the flag error but the command still counted as a success:
a piped weed shell run exited 0, the master maintenance loop logged no
error, and admin script jobs reported the line as done.
Return the parse error as s3.bucket.create (#12544) and the other shell
commands already do. TestCommandsReturnFlagParseErrors runs every
registered command with an unknown flag and fails for the 48 on master.
Signed-off-by: anant <170962632+antcybersec@users.noreply.github.com>
* shell: don't report -h as a failure
With the parse errors returned, -h on those commands now returns
flag.ErrHelp, so the shell printed "error: flag: help requested" and a
piped run exited 1. The flag package has already printed the usage, so
processEachCmd now treats flag.ErrHelp as success for every command.
Signed-off-by: anant <170962632+antcybersec@users.noreply.github.com>
* shell: keep the -h test beside the flag parse test
TestProcessEachCmdTreatsHelpAsSuccess covers the same change as
TestCommandsReturnFlagParseErrors, so it lives in the same new file and
shell_liner_test.go stays as it was.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Signed-off-by: anant <170962632+antcybersec@users.noreply.github.com>
Co-authored-by: anant <170962632+antcybersec@users.noreply.github.com>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* credential: migrate the legacy IAM file without overwriting newer entries
The migration wrote every identity, service account and group from the
snapshot the load read, over whatever the per-entry files held by then, so
a revoke or delete landing in between was undone for good. It now migrates
only the legacy file's own entries, creates each per-entry file only when
none exists, and waits for the next load when the legacy file changed
after the snapshot.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* credential: test changes landing between the IAM load and its migration
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* credential: keep a per-entry file deleted after the IAM load deleted
An entry the load read from its own file is already migrated. When that
file is removed before the migration runs, the legacy copy must not
recreate it.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* credential: test a deleted per-entry file the legacy file also holds
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* s3tables: create a missing directory exclusively in ensureDirectory
ensureDirectory looked the path up and, when it was missing, created it with a
plain CreateEntry. A table create that lost the lookup race then replaced the
directory another create had made, claimed and confirmed since, erasing that
table's record, so a third create could claim the name and win as well.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* iceberg: create missing metadata parent directories exclusively
saveMetadataBlob created each missing directory on the way to the metadata
directory with a plain CreateEntry after a lookup. A table directory another
create had made and written its record onto since the lookup was replaced by a
bare directory, erasing that record.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* iceberg: create missing stage-create marker directories exclusively
The marker path starts at the table bucket directory, which carries the bucket
record; a plain create after a missed lookup replaces a directory another writer
made in between, with whatever it carries.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Volume layouts are keyed by ttl.String(), so a nil ttl and the empty ttl
share one layout. ttlEqual told them apart, so a replica reported with a nil
ttl into a layout created with the empty ttl matched its own layout as a
duplicate, and every copy was quarantined.
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* shell: replicate the downloaded volume before deleting its remote object
volume.tier.download downloaded every replica from the remote tier and let
the last one in the master's location list delete the shared object, so a
replica missing from that list kept reading a deleted object.
Download onto one replica, copy it over every other remote-backed replica,
re-read the topology so replicas that show up meanwhile are replaced too,
and only then delete the last remote-backed replica together with the
object before copying the local volume onto it.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* shell: test tier download replicates before deleting the remote object
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* shell: copy the downloaded volume onto each replica's own disk type
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* shell: test tier download keeps each replica's disk type
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* shell: name the hdd disk type when copying a downloaded volume
VolumeCopy treats an empty disk type as the source's, so an HDD replica
copied from an SSD source was placed on SSD or failed.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* shell: block deletes on the tier download copy source
A readonly-can-delete source took deletes after VolumeCopy read its
index, so the replaced replica lost them and served the files again.
Hard-mark the source readonly while replicas are replaced, then restore
the writable mark or reload the persisted state.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* shell: keep the remote object while replicas are unlisted
Tier download deleted the shared object once the listed replicas were
local, even when the replica placement expected more copies than the
topology listed. Keep it in that case so a rerun can finish later.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* shell: fail tier download when no replica is listed
Also point the final-replica delete error at volume.fix.replication.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* shell: keep the remote object when a server lists the volume twice
A stale twin on another disk of a replaced server still references the
object but is filtered out with its server.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* shell: restore the tier download source when its readonly mark fails
VolumeMarkReadonly blocks local writes and deletes before its second master
notification, so a failed notification returns an error with the source
already frozen. Restore it whenever the status was read.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* shell: report a failed tier download source restore
A restore that leaves the source unmounted or frozen was only printed, so the
command still returned success. Join the restore error into the result.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* shell: wrap the tier download restore error
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* volume_move: persist the move's source freeze across reloads
A live move froze the source in memory only, so an unmount/mount or a
volume server restart mid-move reopened it for writes that the move then
deletes with the source. Persist the freeze when the move froze a
writable source, and reopen the target right after the copy since the
copied .vif carries the freeze. An already-readonly source keeps an
in-memory mark, and a failed move still restores the source writable.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* test: live move keeps the source frozen across a reload, on Go and Rust volume servers
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* test: require 409 from the reloaded frozen source
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* volume_move: reopen an interrupted move's source only without a target copy
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* mount: merge a refused content flush onto the stored entry
A content flush conditioned on the handle's base failed with EIO whenever a
peer had changed the file since, so two mounts appending to one file could
not both close it cleanly. On a refused condition, reread the entry and
replay this handle's changes since its base onto it: the chunks it added or
dropped, and the attributes, extended keys and inline content it changed.
A local truncation keeps its size and clips the peer's chunks to it;
otherwise the larger size wins. The merged entry is published against the
stored one, up to three times, and installed in the handle so its next
flush keeps the peer's chunks. A replaced or deleted file is still refused.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* mount: merge a peer's content change into a dirty handle
A subscription event carrying a peer's content change replaced a dirty
handle's entry and discarded its unflushed pages, so the handle's writes
were lost and its close still returned OK. Fold the peer's entry into the
handle with the same merge a refused flush uses, keep the dirty pages, and
take the peer's entry as the new base. A replaced file still invalidates.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* mount: stop a flush merge once a filer accepts the write
withFilerClient can call the callback on several filers. A refusal from
one left its stored entry behind, so a later filer's success went on to
another merge round.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* mount: refuse a merge that puts inline content next to chunks
A read inside the inline bytes returns them without looking at chunks,
so one side's inline content merged with the other side's chunks hid
data after a successful flush.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* mount: refuse to merge a whole-file rewrite over a peer's chunk change
maybeMergeChunks re-uploads the handle's whole view as unstamped chunks.
Merged after a peer's change, they tie with a peer's own rewrite or bring
back a range the peer truncated, so that flush fails as before.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* mount: clip peer chunks at the lowest truncate, not the final size
A handle that truncated and then wrote past the old end ends up larger
than its base, so the merge did not see the truncate and kept the peer's
chunks inside the range the truncate cleared. SetAttr now records the
lowest size truncated to over the current base.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* mount: keep a peer's truncate when the handle left the size alone
The merge took the larger of the handle's and the stored size, so a peer
truncate came back as a zero-filled tail when the handle only overwrote
bytes inside it. The stored size now stands unless the handle changed
its size or wrote past it, and rebasing keeps only the growth writes made
after the flush was taken.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* mount: retry a read view whose merged manifests did not resolve
A failed SetChunks leaves the chunk group answering every read with the
resolution error, and nothing re-ran it while the entry stayed the same.
The handle now marks such a view (on rebase and on SetEntry) and the
next read or lseek resolves the entry's chunks again.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* mount: keep a dirty handle's writes when a peer change cannot merge
The new merge refusals sent a peer event on the same file down the old
path, which discarded the dirty pages and adopted the peer's entry, so
close returned OK without the writes. The handle now keeps its writes and
base, and the flush meets the conflict and fails.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* mount: keep an unpublished truncate across a peer's base change
The truncate mark was tied to the base pointer, so a peer event that moved
a dirty handle's base (a merged content change or a metadata-only change)
dropped it, and a later merge on a regrown file kept peer chunks inside
the truncated range. Those base moves now carry the mark; a flush ack or a
discarded handle state still clears it.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* mount: wait for the flush before merging a shrink over buffered writes
A write inside the file leaves its bytes in dirty pages without moving
the size. Merging a peer's truncate then adopted the smaller size, and the
chunk uploaded later sat past it, unreadable through the handle. The event
path and the periodic metadata flush now refuse a merge that shrinks the
handle; the close flush uploads the pages first and merges then.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* mount: tell a shortened base chunk from a whole-file rewrite
A truncate shortens a base chunk in place, so it gets a new reference
key, and an old chunk can carry no timestamp. The rewrite refusal now
applies only to unstamped chunks whose file id is new to the base.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* mount: check the creation time when the base has no inode
An entry without an inode or hard link id matched any replacement at the
same path, so a stale handle could merge into a different file. Without
an inode the merge now needs the same creation time. The filer keeps it
across updates; only a synthesized base has neither and still merges.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* mount: keep a dirty handle's writes when a peer replaces the file
A peer entry with another inode at the handle's path went down the
fallback that destroys the dirty pages and installs the replacement, so
close returned OK without the writes. The handle now keeps its writes
and base like it does for an unmergeable change, and its flush refuses to
update the replacement.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* mount: build the content flush tests on freebsd
The invalidation tests, which every platform compiles, now use the
content flush helpers, so the freebsd cross-compile vet failed with
undefined: flushConditionEntry. Nothing in the file is freebsd-specific.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* mount: keep a dirty handle's base when a same-content file replaces it
A replacement with the base's chunks but another inode counted as a
metadata-only change, so the dirty handle adopted it as its base and the
next conditional flush overwrote the new file.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* mount: bring a peer's metadata change into a dirty handle
A metadata-only event moved a dirty handle's base but left its live
entry, so the next flush read the peer's chmod as a local edit and
published the old mode back. Replay the handle's own changes onto the
peer's entry instead, as for a content change.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* mount: wait for the flush before merging peer inline content over buffered writes
A buffered write is not a chunk yet, so a merge could install a peer's
inline content that later reads return ahead of the write's chunk.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* mount: track a whole-file chunk rewrite on the handle
The merge told a rewrite by its unstamped chunks, which a manifest fold
hides: manifests are unstamped either way. Record the rewrite against the
base it was made from, carried across peer base changes like a pending
truncate, and refuse the merge from that.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* mount: refuse a merge that shortens a manifest at the truncate
Clipping a manifest reference leaves its members whole, so peer bytes past
the handle's truncate could reappear once the file grew again.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* mount: flush a dirty handle after a peer replaces its file
The replacement test checked only the retained handle state. It now
flushes against a filer holding the replacement, including one with the
base's content, and checks the flush fails and leaves it unchanged.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* mount: test an append fsync after a peer's attribute-only change
A second mount's acknowledged chmod, chown, utimes or setxattr leaves
the content alone, so the writer's conditional content flush must still
publish its append and keep that change. On master the refused flush
surfaces as EIO from Fsync.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* s3api: sign every operation subresource in signature V2
The V2 canonical resource only kept the subresources inherited from the
original allowlist, so tagging, retention, legal-hold, cors, encryption,
object-lock and the other keys the router dispatches on were left out of
the signature. A V2 signature issued for one request therefore also
verified that request with one of those keys appended, running a
different operation under the signer's identity.
Add every operation-selecting subresource to the signed list. Clients
that already sign tagging, cors, object-lock or replication in V2 now
verify as well.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3api: sign every value of a repeated signature V2 subresource
canonicalizedResourceV2 kept only the last value of a repeated query key,
while handlers read the first. A request signed for versionId=A could be
sent as versionId=B&versionId=A and act on version B. Sign every value in
order, as V2 clients do, so a second value no longer verifies.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* fix(mount): preserve acknowledged chunks during attribute updates
* mount: keep the WFS field block aligned
Adding the publication-lock fields inside the aligned block made gofmt
re-indent 25 unrelated lines. Put them in their own group after
dirIdleEvict, matching the enterprise layout, so syncs do not conflict there.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* mount: finish attribute saves through a FUSE interrupt
saveEntryAttributes derived its RPC context from the syscall's cancel
channel, so an interrupt while the conditional UpdateEntry was at the
owner returned EIO for an update the owner then applied, and skipped the
local install and retention. Keep the metadata deadline but drop the
cancellation, like the other metadata mutations.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* mount: map attribute CAS ids in place instead of cloning the entry
saveEntryAttributes cloned the whole entry three times per attempt only
to rewrite uid/gid for the filer, and each clone is as large as the
chunk list. Switch the ids in place and restore the exact local values
after the call, keeping one clone. The test now checks the reply carries
the local uid.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* mount: compare attribute updates against the filer's own uid/gid
With -map.uid/-map.gid, a filer id that is also a mapped local id reads
back unchanged and maps forward to another filer id. The IF_ENTRY_EQUAL
expected entry then never matched, so chmod, utimes and xattr changes on
such files failed with EAGAIN after three attempts, and an update would
have rewritten the owner. Keep the filer ids read with the entry for the
ids the syscall leaves alone.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* fix(mount): align attribute retention with read lock order
* mount: take the handle lock before entryLock in the attribute snapshot
attributeEntrySnapshot held entryLock while it queued for the shared
handle lock. A Read holding that lock waits on entryLock, the snapshot
waits behind a queued exclusive (an invalidation or a Write), and the
exclusive waits on the Read.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* fix(mount): complete handle lock order and preserve explicit ID intent
* mount: re-read a refused attribute update from the file's owner filer
The filer checks the IF_ENTRY_EQUAL condition on the owner of the path's
write route key. The retry read any filer, so behind one that had not
applied the owner's latest update, every attempt sent the same stale base
and the syscall failed with EAGAIN. Read from the owner once the lock ring
is known, like lookupFilerEntry, but keep the filer's raw ids.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* fix(mount): condition content flushes on the observed entry
* mount: stop retrying a flush the filer refused on its condition
The filer reports a failed CreateEntry condition as FilerError_PRECONDITION_FAILED
in the response, which surfaces as filer_pb.ErrPreconditionFailed rather than a
FailedPrecondition status. The flush retry loop treated it as transient and
replayed the refused flush three more times over 7 s before returning EIO.
The test filer now answers like the real one.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* mount: take the handle base from the entry the filer stored
After an acknowledged flush the handle base was the requested entry, but the
filer stores it with a fresh commit_ts_ns (and fills crtime, inode, atime), so
the next IF_ENTRY_EQUAL flush on the same handle never matched: write, fsync,
write, fsync returned EIO, as did close after a periodic metadata flush and the
first flush after an eager create. Use the stored entry from the
acknowledgement's metadata event.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* mount: condition the periodic metadata flush like the close flush
The periodic metadata flush of an open, dirty handle sent the same whole entry
through an unconditional CreateEntry, so a file held open past the flush
interval still overwrote a peer's acknowledged append, and the close flush then
matched the overwritten state. Both paths now publish through one helper that
carries the handle's write condition.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* mount: republish a refused flush once when the stored entry is unchanged
A handle's base can differ from the stored entry without any peer write: a
rename while the file is open restamps the entry under its new name, and with
-map.uid a filer id that is also a mapped local id does not map back to itself.
Each made every flush of the handle fail with EIO. On a miss, read the stored
entry and, if it still holds the observed state apart from name, commit stamp,
id encoding and derived size, retry once with that entry as the condition.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* mount: take the filer's entry once a write open holds the DLM lock
The lock orders this writer after the previous holder, but the holder's
flush may not have reached the local cache yet, so the handle's base
named the entry from before the wait and the conditional flush refused.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* mount: share the open-existing path of Create
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* mount: open the file a previous DLM holder created during a deferred create
A deferred create never asks the filer, so a create that waited for the
lock went on to publish over a file the previous holder had just made,
and its flush expecting no entry was refused.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* master: send the lock ring snapshot to mounts
A mount running -dlm is a lock client like the S3 gateway, and needs the
ring on connect to reach a file's owner filer before the next membership
change is broadcast.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* mount: follow the masters' lock ring under -dlm
The lock client only knew the seed filer, so every lock took the forward
hop and the mount had no way to tell which filer owns a file. Subscribe to
the masters named by the filer, like the S3 gateway, and wait briefly for
the first ring before serving.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* mount: read the DLM-ordered entry from the file's owner filer
The filer checks a flush's condition on the owner of the path's write
route key, and another filer may not have applied the previous holder's
flush yet. Look the entry up on that owner once the ring is known.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* volume(rust): untrack queued writes before replying
The durable write worker sent each write's result and only then dropped
the batch's pending-write trackers, so a caller could see its write
finish while the write was still tracked as pending. Go untracks in the
caller before writeNeedle2 returns. Drop the trackers before sending the
results so the Rust queue path matches.
test_replicate_delete_cancels_a_queued_write asserted the count was zero
right after the fsync write returned and failed about 1 run in 100 under
load (Rust Unit Tests on master c2625dd).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* volume(rust): untrack a write rejected for a remounted volume before replying
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* volume: skip VolumeDelete when its RPC was cancelled before the delete
A caller that cancels (a lost tiering lease, an expired deadline) still
had its admitted delete run in the Go server, including one that waited
behind the copy guard. The Rust server drops a cancelled handler, so
return the context error before the delete starts.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* volume: wait until the queued delete blocks on the guard before cancelling
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* s3api: verify signed request bodies against the signed payload hash
A SigV4 signature covers the declared x-amz-content-sha256, but only
getRequestDataReader (PutObject, UploadPart) compared the body with it.
Every other handler parsed the body as received, so a signed request,
DeleteObjects for example, could be replayed with a different body.
After a V4 signature verifies, wrap the body so it is handed out only
once all of it has been read and matched against the hash the signature
covered; a decoder that stops reading early never sees unverified bytes.
getRequestDataReader unwraps it and keeps verifying uploads as they
stream.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3api: answer XAmzContentSHA256Mismatch for refused DeleteObjects bodies
DeleteObjects answered a body that failed its payload-hash check with
InternalError and CompleteMultipartUpload with MalformedXML. Name the
check that failed instead, as PutObject does.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3api: check Content-MD5 on signed request bodies
Signature V2 signs Content-MD5 rather than a payload hash, and a V4
request sent with UNSIGNED-PAYLOAD may still sign it, but bodies read
outside the upload path never compared it with the body. Check it with
the payload hash once a V2 or V4 signature verifies: BadDigest on a
mismatch, InvalidDigest when it is malformed.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3api: verify presigned uploads against a signed payload hash
A presigned V4 URL can sign X-Amz-Content-Sha256 as a query parameter,
but getRequestDataReader compared upload bodies only with the header of
that name, so the signed hash went unchecked. Verify uploads against the
hash the signature covered.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3api: check Content-MD5 on bodies buffered for their payload hash
IAM and other non-S3 requests signed without x-amz-content-sha256 have
their body buffered and hashed during signature verification. That
buffer counted as already bound, so their Content-MD5 was never checked.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3api: check Content-MD5 on directory marker bodies
A PUT of a key ending in "/" stores its body on the directory entry
without going through putToFiler, so its Content-MD5 was never compared
with the body. Answer BadDigest on a mismatch, as putToFiler does.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3api: wrap and unwrap signed bodies through helpers
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* shell: confirm trim survivors still serve the volume before deleting
volume.fix.replication compared the doomed replica's index with every
survivor and then deleted it unconditionally. A survivor that went away
while the indexes were read still let the delete through, leaving fewer
copies than the check had counted. Re-read each survivor's VolumeStatus
after the index checks and skip the trim if any no longer has the volume.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* shell: test a trim whose survivor goes away after its index read
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* s3: ignore unsigned object headers on POST Object
The POST policy signs form fields only, but the upload read tagging,
user metadata, storage class, object lock and the stored content
headers from the request headers as well. Drop them before the form
fields are applied.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3: test POST Object request headers against signed form fields
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* s3api: answer PutObjectLegalHold on a missing key with NoSuchKey
In a versioned bucket a key with no latest version resolves to
ErrLatestVersionNotFound, which the legal hold handler did not map, so
the request failed with 500 InternalError. PutObjectRetention already
maps it to NoSuchKey.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3api: keep failed reads distinct when resolving an object lock target
getObjectEntryForUpdate turned every lookup error into a not-found
sentinel, so a failed filer read made PutObjectRetention and
PutObjectLegalHold answer NoSuchKey for an object that exists. Only a
genuine miss becomes the sentinel now, matching getObjectEntry.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3api: answer object lock calls on a delete marker with NoSuchKey
PutObjectLegalHold and PutObjectRetention on a key whose current
version is a delete marker returned 200 and wrote the lock onto the
marker, and the matching GETs read it back. A key whose current version
is a delete marker does not exist, and delete markers take no lock, so
these calls now answer NoSuchKey like the tagging and ACL handlers.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3api: let a lock recorded on a delete marker not block its removal
Delete markers are not WORM-protected, but a legal hold or retention
written onto one (possible before the previous change) made both the
gateway check and the routed delete's filer-side WORM guard refuse to
remove the marker, so the object could not be restored and lifecycle
could not clean the expired marker up.
Lock enforcement now ignores a lock on a delete marker, and the routed
version deletes in DeleteObject and lifecycle drop the WORM guard when
the version they read is a marker. A version id never changes kind, so
the marker read is the one deleted; the null version keeps its guard
because a PUT can replace it.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
The master, volume, volume-rust, server, mini, filer and s3 cases and the
fallback forwarded the user arguments as an unquoted $@, so the shell split
each one at whitespace and globbed it against /data. An -admin.password with
a space reached weed as two words, and weed ignored every flag after the
stray word. Quote "$@" as the other cases already do; $ARGS stays unquoted
since the script builds it from its own words.
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* replication: let the local sink check a delete against the live source
LocalSink removed the destination path for any delete it was handed. A
delete that reaches the sink after the source recreated the path (a peer
filer's event delivered late, or a historical event replayed) removed the
newer file, and no later event brings it back.
DeleteEntryVersion receives the deleted entry. Before removing an existing
destination, the local sink looks up the source path and keeps the
destination when the source holds an entry of the same kind committed
after the deleted version (commit_ts_ns, or mtime for unstamped entries).
A live entry that is the deleted version itself or older means the source
store lags the event, so the delete proceeds. A failed lookup returns the
error so the event is retried instead of deleting a file that may be live.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* filer.sync, filer.replicate: hand the deleted version to the sink
Pure deletes, renames without a native move and moves out of the
watched directory now go through sink.DeleteEntryVersion with the old
entry, so a sink that checks the live source (the local sink) can keep a
path the source recreated after the deleted version. Other sinks fall
back to DeleteEntry unchanged.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* replication: order the local sink check by the delete's mutation time
A source filer can hold a version written concurrently with the delete:
newer than the deleted entry but older than the delete itself. The
source's own replay applies the delete over it, so keeping the backup
copy because it outranks the deleted entry left a file the source no
longer has.
The sink now gets the delete's mutation time (MutationTsNs, else TsNs)
and keeps the destination only when the live entry was committed after
it, the order Filer.Replay keeps. That also covers a stamped replacement
of an unstamped entry whose mtime did not move. filer.replicate events
carry no time, so they keep the deleted entry's stamp as the bound, and
mtime only when nothing is stamped.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* replication: count an unstamped live entry from its mtime
A source filer that predates commit stamps stores entries with no
CommitTsNs, so comparing the live stamp with the delete's time never
kept a replacement there. An unstamped live entry now counts from its
mtime against the delete's time.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* replication: keep a stamped live entry when the deleted one is unstamped and the delete has no time
filer.replicate gets no delete time from the notification queue. When the
deleted entry has no commit stamp either, a stamped live entry was written
after it, so its mtime no longer decides.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
The line parser only trimmed quotes, so `key = "abc\n"` gave the Rust
volume server the bytes `abc\n` with a literal backslash while the Go
servers sign with a newline, and every JWT failed. Literal strings kept
their quotes and a trailing comment stayed part of the value.
Decode each value with the TOML parser; text it rejects keeps the old
quote trimming.
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* operation: probe octet-stream uploads for gzip like untyped ones
The 128-byte gzip probe ran only when the caller passed no MimeType, so an
upload labelled application/octet-stream or binary/octet-stream was never
compressed. S3 PutObject and UploadPart pass the request Content-Type (or the
octet-stream that mimeDetect fills in), so every binary S3 upload skipped
compression that the same bytes get through the filer.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* operation: check the probed upload keeps the caller's content type
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* wdclient: keep a shared volume lookup alive when its first caller leaves
Concurrent misses for the same volumes share one provider lookup, but it
ran on the first caller's context, so that caller's cancellation failed
every request waiting on it. The lookup now runs on a context no single
caller can cancel, and each caller stops waiting when its own context ends.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* wdclient: wait for the second caller to join before canceling the first
A fixed sleep did not prove the second caller had joined the shared lookup:
a late caller could hit the filled cache and pass without waiting. The test
now waits until the caller reads its own Done channel, which happens only
after DoChan registered it.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* filer: let the fake-master test helper serve any master
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* filer: gzip metadata log pieces before uploading them
Every metadata event carries the old and new entry with their whole chunk
lists, so a file grown by small fsynced appends logs the same chunk records
again and again: 2000 appends of 256 bytes through weed mount wrote 34.8 MB
of log. The uploader only compresses when the first 128 bytes compress, and
a log piece starts with fresh file ids and ETags, so it stored the log as is.
The filer now gzips each piece itself and keeps the compressed bytes when
they save at least 10%. After the same workload the volumes hold 3.05 MB
instead of 35.2 MB, file data included. Readers already honor the chunk's
IsCompressed flag on every path (filer-side decode, log file refs read
straight from volume servers), and the assign request sizes the bytes
actually stored.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* mount: keep the filer status in the flush create error
flushMetadataToFiler formatted the CreateEntry error with %v, so callers
classifying it saw only the message, which includes the file path. Wrap
it so the gRPC status the filer sent stays reachable.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* mount: retry a writeback metadata flush until the filer answers
With -writebackCache, close() returns before the chunk references reach
the filer. The deferred flush gave up after four attempts, about a minute
once each attempt's own filer retries are counted, then released the
handle: a longer filer outage dropped the file even though write() and
close() had succeeded, and its uploaded chunks became orphans.
Keep retrying for as long as the filer is unreachable, as the writeback
chunk uploads already do. A filer that answers with an error still ends
the flush after the bounded attempts, and an unlink of the draining file
still ends it on the next attempt.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* mount: retry only an unanswered filer call without a bound
The unbounded phase used util.IsTransientError, which falls back to
matching the error text. A refusal the filer sends in its response
(read-only, name too long, a plain error string) has no gRPC status, and
the flush error carries the file path, so a file named like
"unavailable-transport.log" kept retrying against a filer that answered.
Retry without a bound only when the call failed with a gRPC Unavailable
status: the dial, the connection or the mount's call deadline gave out.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* s3api: deny signed reads that name a version file by its physical path
A stored version lives at <key>.versions/v_<id>, and that path is also a
valid object key. Anonymous reads already refuse it; signed GET, HEAD,
copy sources and object subresources did not, so a caller without
s3:GetObjectVersion, or under a Deny on the logical key, could read any
version through it. Apply the same check in the shared entry recheck.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3api: recheck a version file key before answering a conditional read
A conditional GET or HEAD answered 304 or 412 with the ETag before the
handler's entry recheck, so the physical version path still confirmed a
version's ETag to signed callers.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3api: look up version file keys in the lookup-based policy recheck
Retention, legal hold, tagging writes and deletes only fetched the entry
for the recheck when the bucket had a policy, so without one they still
acted on a version file named by its physical path. Fetch it whenever
the key has the version file shape.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3api: refuse a directory-marker delete that names a version file
A trailing slash kept "<key>.versions/v_<id>/" out of the version file
shape check, and the lookup recheck misses the entry for that form, so a
DELETE reached deleteDirectoryMarker and removed a chunkless version or
delete marker without s3:DeleteObjectVersion.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
The ring tests used 127.0.0.1:18889 as an unreachable peer, but its gRPC
port 28889 is a common filer port: on a shared CI host a live filer there
accepted the forwarded exclusive create, and
TestCreateEntryExclusiveLeavesNonOwner failed. The peer now listens on a
port the test owns and drops every connection.
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* vector: share one Lance session across reads
Each request opened its table in a session of its own, so every query
loaded and decoded the index again. Get, list and query now open in one
session the server keeps, sized by -index-cache-mb and
-metadata-cache-mb; writes keep a session of their own.
Lance keys some of its cached entries, such as a version's index list,
by location and version alone, and an index dropped and created again at
its location numbers its versions from 1 again: a shared session answered
a query on the new table with the old table's index. CreateIndex now
marks each table with an id in its schema metadata, and the session
starts over when a location turns up holding another table.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* vector: share a version lease among the reads of that version
Every get, list and query tagged the version it read and deleted the tag
afterwards: two filer writes and a manifest read per request. Reads of
one version now share one tag, which goes 10 s after the last read took
it; a tag is reused for at most 5 minutes, so a busy version never
reaches the reapers' TTLs.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* vector: keep opened tables for the next read
A read built the table's store and read its latest manifest each time.
Reads now keep the opened table by location and encryption, up to 1024,
and move it to the latest version, which lists the versions and takes
the manifest from the session. A handle opened in an earlier session is
not reused.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* vector: key shared leases by manifest, and take one per version at once
A table dropped and created again within the linger reused, for a
version number the old table had, a lease whose tag the drop had
deleted. Leases are now keyed by the manifest's ETag as well. Reads that
arrive before a version has a lease wait for the first to take it,
rather than each writing a tag.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* vector: drop kept tables when the shared session starts over
A kept table holds the session it was opened in, so a replaced session
and its caches lived on in the handles of other tables until they were
evicted.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* vector: prune shared lease slots only when they double
Every lease lookup scanned all slots under the map's lock, a cost that
grew with the versions read in the last linger.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* vector: shrink the lease slot map after a prune
After a burst the map kept its peak allocation, which every later prune
scanned.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* vector: keep a lease slot a read has taken through a prune
A read that took a slot and had not yet locked it could see the slot
pruned, and then pin through it while the next read pinned through a
new slot, writing a second tag.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* admin: restyle the UI as a calm console
Grey canvas with white sheets, a docked sidebar that marks the current
page with an indigo tint, ink primary buttons, tinted status pills,
dense tables with tabular numbers and spectrum capacity meters. The
styles are design tokens layered over Bootstrap's own variables, with
light and dark values, so the page templates stay unchanged.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* admin: add a light, dark and automatic theme switch
The choice is kept in localStorage and applied before the page paints,
on the sign-in page too; automatic follows the system setting.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* admin: drop hard-coded colors from page templates
The volume id link, utilization track and plugin form borders used
fixed light-theme colors; sparklines now use the data palette.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* admin: theme the plugin job detail panel background
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* admin: keep the selected radio dot visible in dark mode
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* admin: cycle the theme from memory when storage is blocked
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* admin: keep success progress bars green
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* admin: keep keyboard focus visible on the toolbar togglers
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* admin: use contrasting text on danger and success buttons in dark mode
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* admin: lead the dashboard with cluster health and one row of tiles
The header now says in one line whether the cluster is healthy and how
many masters, volume servers and filers report. The two rows of stat
tiles become one: the volume size limit and EC volume count move into
footnotes, the mount client card joins the cluster counts, and the
counts link to their pages instead of sitting in nested cards. The
duplicate version list and the bucket shortcut in the header are gone,
and the section menus become plain View all links.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* admin: export the user columns the table actually has
The users table has username, email and access key columns, but the
export waited for six cells per row and so always produced only the
header line.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* admin: drop table titles that repeat the page title
List pages titled their only table with the page title again. The
table now sits right under the summary tiles, and the export actions
that were hidden in the table header menu move to the page header as
an Export button, matching the cluster pages. The policies menu linked
to nothing and is removed.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* admin: drop clock and duplicate summary tiles
Buckets, users, policies and service accounts spent a summary tile on
the refresh time, which the page footer already shows (policies gains
that footer). The users page also showed its user count twice.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* admin: show unset bucket settings as a dash
Rows of No owner, No quota and Not configured made the bucket table
hard to scan. Unset cells now show a quiet dash that keeps the wording
as a tooltip and for screen readers, and the name and creation date no
longer wrap (the full timestamp is in the date's tooltip).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* admin: export buckets through the page's API exporter
admin.js loads after the page, so its copy of exportBucketList replaced
the buckets page's own one. That copy scraped the visible rows, so it
exported only the current page, without CSV quoting, and picked up the
display text of the cells. The page's exporter reads every bucket from
the API with full timestamps and escaped fields.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* admin: escape quotes in the user export
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* admin: download the bucket export as a Blob
The exporter put the CSV into a data URL with encodeURI, which leaves #
alone, so a bucket owner containing # ended the download there. Use the
shared downloadCSV helper instead.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* admin: count masters as configured, not reporting
The master list comes from raft membership, which keeps a follower
that is down, so it does not show which masters are reporting.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* admin: link the Prometheus metrics endpoint from the sidebar
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* admin: drop the sidebar metrics link
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* s3api: record the caller on AssumeRole sessions
The session name is the caller's choice; the session now carries the
caller itself as the user it was issued to, as the web identity, LDAP
and credential paths already do.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3api: take a role session's aws:username from its user, not its name
An assumed-role ARN ends in the session name the caller picked, so
naming a session after another user resolved ${aws:username} and
${aws:userid} to that user in bucket and attached policies. Resolve
them from the user the session was issued to instead.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* iam: take a role session's aws:username from its user, not its name
IsActionAllowed set aws:username to the last segment of an assumed-role
ARN, the caller-chosen session name, for role and bucket policies. Use
the user the session was issued to, and drop a forwarded value when the
session names none.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3api: pass a role session caller's user on to the session it assumes
A role session that calls AssumeRole is named by its session ID, so the
new session recorded that ID as its user. An STS session caller now
passes on its own recorded user, or none.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3api: size the session claims map from the request context
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* admin: take master HTTP ports from raft member names
The masters page and dashboard turned each raft member's gRPC address
into an HTTP address by subtracting 10000 from the port, so masters with
a custom -port.grpc showed wrong or negative ports. Raft members are
named after their master address, which carries the HTTP port; use it
and keep the old derivation only for names that are not addresses.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* admin: test raft member address mapping
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* s3api: drop cached gRPC channels when a fake filer takes a port
A test that dials a dead owner leaves its failed channel in the
process-wide gRPC connection cache. When a later test's fake filer is
given the same port, its first RPC fails at once with the cached
connection refused, which is how TestRoutedMultipartFinalize failed on
the 32-bit job. The fake filer helpers now share fakeFilerAddress, which
invalidates any cached channel for the new address.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3api: test a fake filer on a dead owner's reused port
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3api: drop cached channels for dead test filer addresses too
closedFilerAddress can be handed a port whose cached channel belonged to
a fake filer that just stopped and may still look ready; then the dead
owner's transaction is sent on it instead of failing over. Build its
address with fakeFilerAddress as well.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* s3api: split upload ACL assembly out of preparePutObjectACL
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3api: store POST Object acl and grant fields on the object
The form acl and x-amz-grant-* fields were authorized but never assembled,
so the object kept no ACL: acl=private on a public-read bucket stayed
anonymously readable and invalid values were accepted. Build them with the
same rules as PutObject, which also covers the Block Public Access check.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3api: test POST Object stores its upload ACL
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3api: take the POST Object ACL from the signed form only
ACL request headers are not covered by the POST policy, so a holder of a
signed form could add one and have it stored with the upload.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3api: reject a POST acl field the policy does not cover
The acl form field now sets the stored object ACL, so like the x-amz-*
fields it needs a policy condition, as AWS requires.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3api: test unsigned ACL headers and uncovered acl fields on POST
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3api: evaluate POST upload policies on the canonical ACL
Policy conditions on the grant headers saw the raw form value, so an
equivalent spelling of a denied grantee passed. Use the PutObject request
shape that the PUT path evaluates.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3api: check bucket policy for POST ACLs with auth disabled
PutObject applies the bucket policy to an explicit upload ACL when IAM is
off; POST now does the same before storing it.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3api: reject differing acl and x-amz-acl POST fields
Both promote to the canned ACL header, so map order picked the stored ACL.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3api: test POST grant canonicalization, auth-off ACL policy, acl conflicts
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3api: check the auth-off POST ACL policy after loading the bucket
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3api: reject an empty acl or x-amz-acl POST field beside a differing one
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3api: test the auth-off POST ACL policy on a cold bucket cache and empty acl aliases
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3api: load the bucket before per-key object authorization
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3api: test the POST ACL policy on a cold bucket cache with auth on
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3api: stop per-key object authorization when the bucket fails to load
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3api: test the cold-bucket POST ACL allow path and a failed bucket load
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* lance: require UpdateTable to deregister a table
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* lance: require UpdateTable to register over an existing table
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* lance: test that GetTable alone cannot deregister or re-register a table
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* lance: a register that finds a table the caller cannot read needs UpdateTable to bring it back
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* lance: decide declare write access by whether the table existed, not by what the caller can read
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* lance: test register and declare by a caller that cannot read the table
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* lance: register clears the marker on the create branch only after checking it
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* lance: keep a register whose table was created when the marker check fails
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* filer/leveldb_handle: reopen a database whose journal latched a write error
goleveldb keeps the first journal write error and returns it for every
later write until the database is reopened, so one transient ENOSPC or
EIO fails all writes until a restart. The handle offers the leveldb.DB
methods the stores use; a write that fails on an OS error probes the
storage and, once it accepts writes and no call holds the database,
closes and reopens it and retries the write once.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* filer: open the leveldb stores through leveldb_handle
leveldb, leveldb2 and leveldb3 now recover from a transient storage
write error without a restart.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* filer: leveldb stores return iterator errors from listings and deletes
A listing on a database that cannot be read reported an empty
directory, and DeleteFolderChildren wrote an empty batch, because the
iterator error was never checked.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* filer/leveldb_handle: test a reopen that fails after the probe passes
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* filer/leveldb_handle: probe storage before draining readers
A probe that stalls in Create, Write or Sync held mu exclusively and
blocked every read on the handle. Reopen attempts are now serialized by
their own mutex, and the probe runs before the drain.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* filer/leveldb_handle: hold new calls back while a reopen drains
TryLock succeeds only at an instant with no reader, so overlapping
listings could keep a reopen from ever draining while each write spun
for the full drain timeout. The drain now holds new calls back for at
most drainTimeout; a call nested inside a running one waits that out
instead of deadlocking. A drain that times out holds off the next
attempt for busyReopenInterval, so new calls are not held back over and
over.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* filer/leveldb_handle: bound the wait for a running reopen
A write that hit the latched error waited on the reopen mutex for as long
as another caller's storage probe ran, so a stalled probe queued every
writer. Callers now wait at most reopenWait and return their error, and
still retry once a concurrent reopen succeeds.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* filer/leveldb_handle: tests wait for the drain and readers explicitly
The nested-read test polled for a drain that could start and end between
polls and then spun forever; it now syncs on the probe, widens the drain
and fails if the put returns first. The overlapping-readers test waits
until every reader holds an iterator before it writes.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* s3: authorize an empty versionId as the current-object action
GET, HEAD and DELETE treat an empty versionId as the current object, but
the action resolver keyed on the parameter's presence and authorized it
as GetObjectVersion/DeleteObjectVersion, so a Deny on GetObject or
DeleteObject did not apply.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3: test an empty versionId against a version-only policy
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3: cover a bare versionId on HEAD
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* volume: refuse a tier upload whose .dat grew during the copy
AttachRemoteDatFile only checked that the data backend was still the one
copied, so a needle write acknowledged after the copy read the .dat was
missing from the remote object, and the upload then switched reads to it
and removed the local .dat. Compare the .dat size with the copy under the
same data-file locks; on a mismatch the handler deletes the copy and keeps
the local file.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* seaweed-volume: refuse a tier upload whose .dat grew during the copy
Mirror the Go AttachRemoteDatFile check. The tier-up attach took the
write lease only inside each step, so a needle write acknowledged after
the upload read the .dat was missing from the object the volume then
switched to, and the local .dat holding it was removed. Hold one write
lease across the instance, revision and size checks, the remote-file
update, the .vif save and the backend swap, as tier-down does; a size
mismatch aborts, deleting the object and keeping the local .dat.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* volume: compare the physical .dat size before a tier attach
DiskFile pads its size past an unaligned tail, which a read-only load
keeps, while the upload copies the physical bytes. Stat the file itself
so such a volume is not refused as written during the copy.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* shell: parallel moves and a per-run limit for volumeServer.evacuate
-maxParallelization runs up to N EC shard moves at once (default 1), with the
ec.balance scheduler: never two shards of one volume at a time, free workers
go to the least busy targets. The moves are planned first with the same target
choice as before, then applied through volume_move.Mover.MoveEcShards, which
verifies the target registered a shard before deleting it on the source.
-movesPerExec limits how many volumes and EC shards one run moves (default 0
for unlimited), like -volumesPerExec in volume.balance, so a big server can be
emptied in several runs.
With -skipNonMoveable a shard that fails to move (a read error on a failing
disk, for example) no longer drops the other shards of its volume; they are
moved and the failures are listed at the end. Without it the run stops at the
first failure as before. Losing the admin lock stops the run in either mode.
Refs #12501
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* shell: plan evacuation again after failed moves; fail on a lost lock
A failed shard move kept the target slot it was planned on, so a readable
shard that found no destination in the plan stayed on the server although
the room was never used. Failed moves now give their targets back to the
model and the shards without a destination are planned again, up to
maxEcEvacuationRounds times; the failed shards are not tried again.
With -skipNonMoveable a lost admin lock was printed as a skipped failure and
the run returned success. ec.ErrLockLost is now exported and always fails
the run.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* shell: volumeServer.evacuate reports -movesPerExec after an applied EC run
The round loop only checked the limit after a failed move, so an applied run
that stopped at -movesPerExec without failures never said to run it again.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* shell: volumeServer.evacuate plans again for as long as failed moves leave room
The ten-round cap left readable shards on the server once more than ten
rounds of unreadable shards had taken the free room first, and a rerun
fails the same shards first again. Every round moves or fails each shard it
plans and a failed shard is not planned again, so the rounds end without it.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* shell: volumeServer.evacuate gives a failed move back every slot it took
When the target already held the shard in another disk type, adding it to
the planned disk type still took a slot, but the undo gave back only the
explicit one. Record the slots the plan took and restore exactly those.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* ec: an evacuation move that landed on its target keeps the target slot
A move can fail after the target registered the shard, when only the source
unmount or delete failed. Evacuate then gave the target's slot back and could
plan another shard onto a full server. The runner now checks the target after
a failed move and leaves a shard the target has out of FailedMoves.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* shell: volumeServer.evacuate gives back the slot of a failed move onto a shard the target had
A target that held the shard before the plan reports it after any failed
move, so the runner took such a move for one that landed and evacuate kept
the slot the plan had taken. The runner now lists those moves in
LandedMoves, and evacuate undoes the ones whose target had the shard, then
plans again with the room they gave back.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
* util: add WaitUntil, a cond wait bounded by a context and a timeout
The caller holds the cond lock. Cancellation and the timeout wake the
waiter through a broadcast taken under the same lock, so no wakeup is lost.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3: bound the wait for an upload slot
A PUT that arrived while the concurrent upload limits were exhausted waited
on the condition variable with no bound. The handler reads nothing from the
body while it waits, so a client that gave up and reset its connection went
unnoticed: the socket and goroutine stayed pinned until a slot freed, and a
stalled volume server grew the gateway's fd count by one per aborted PUT.
The wait now ends on InflightUploadDataTimeout with 503 SlowDown, which S3
SDKs retry, and on request cancellation. An admitted upload takes its slot
under the lock, and a finished upload broadcasts, so freed bytes admit every
waiter that fits instead of one.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* filer: bound the wait for an upload slot
The filer's PUT/POST upload limits queued requests the same unbounded way.
Reuse the bounded wait: reject with 429 after InflightUploadDataTimeout, as
the volume server does, and stop on request cancellation.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* command: add -inflightUploadDataTimeout for the S3 gateway and the filer
weed s3 -inflightUploadDataTimeout, weed filer -inflightUploadDataTimeout and
-s3.inflightUploadDataTimeout, weed server and weed mini
-filer.inflightUploadDataTimeout and -s3.inflightUploadDataTimeout. The
default of 60s matches the volume server's flag of the same name; 0 keeps
the old unbounded wait.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3: reject a queued upload without reading its body
WriteErrorResponse drains the request body before answering. Behind the
stats recorder the drain has no read deadline, and reading the body sends an
Expect: 100-continue client the go-ahead to upload what is about to be
discarded. Answer the rejection with an empty-body copy of the request so
the drain is skipped; net/http closes the connection when a large unread
body remains.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3, filer: release an upload slot under the cond lock
A waiter checks the counters under the cond lock and then waits. A release
that decremented and broadcast without the lock could land between the two,
so the waiter slept with a slot free until the next release or its timeout.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3, filer: close the HTTP/1 connection when rejecting a queued upload
net/http reads up to 256 KiB of an unread request body before sending a
keep-alive reply, so a client that stalls its body held back the 503 or
429 until the connection idle timeout. Connection: close skips that read.
HTTP/2 is left alone: there it would shut down the whole connection.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* shell: s3.clean.uploads deletes uploads over filer gRPC
Same DeleteEntry the HTTP handler runs, without minting a filer JWT,
and the gRPC path is the one a locked, routed delete needs next.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* shell: s3.clean.uploads frees an open upload under the object write lock
The cleaner deleted a stale upload recursively with no lock, so a
CompleteMultipartUpload could pass its upload-exists check after the
part entries and their chunks were gone, and publish an object over
freed chunks. Do what AbortMultipartUpload does: hold the object's
distributed write lock, and free an open upload in an ObjectTransaction
routed to the object's owner filer, conditioned on the object not
carrying the upload; if it does, delete the metadata only.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* shell: s3.clean.uploads keeps the object write lock for the whole cleanup
A large upload's recursive delete can outlast one 15 second lease, and
a completion on the distributed-lock path could then take the lock and
commit over chunks being freed. Renew the lock in the background and
skip the delete once it is lost.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* shell: s3.clean.uploads locks the path a completion commits under
For a trailing-slash key a non-versioned completion writes and locks the
entry inside the directory the key names, not the object path, so the
cleaner's transaction did not serialize with it. Lock and check the
same path the completion uses for the bucket's versioning state; the
route key stays the object's.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3api: abort locks the path a completion commits under
A routed abort locked and checked the object path, but for a
trailing-slash key a non-versioned completion commits under the entry
inside the directory the key names. Use the same path the completion
locks for the bucket's versioning state.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* shell: end s3.clean.uploads bucket failure line with a newline
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* cluster: NewBlockingLongLivedLockWithin bounds the wait on a held lock
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* shell: s3.clean.uploads skips an upload whose object lock stays held
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* cluster: share the contended lock filer setup across lock tests
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* shell: s3.clean.uploads reads the bucket's versioning state per upload
The commit path a routed completion locks depends on versioning for a
trailing-slash key. Reading it once per bucket before the run left a
window as long as the run; read it just before the delete, as the
gateway's abort does.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* mount: remove the local control socket when the mount exits
weed mount listened on /tmp/seaweedfs-mount-<hash>.sock and removed it only
when a later mount of the same directory started, so every unmount or
SIGTERM left one socket file behind. Remove it on the normal return path and
in the interrupt hook, unless a later mount of the same directory has
already replaced it.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* mount: remove the control socket before a failed mount exits
glog.Fatalf and glog.Exitf call os.Exit, which skips the deferred removal,
so a mount that failed after creating its socket (mount point not
available, fusermount failure) still left it behind.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* mount: register the socket cleanup hook before mount setup
A SIGTERM that arrived while the mount was still setting up, for example
waiting on the filer to create the mount root, exited through grace without
running the deferred cleanup. Registering the hook right after the listener
is created makes it run last, after the unmount and filesystem hooks.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* filer/etcd: scan entries from one revision
volume.fsck walked etcd one directory at a time, so a file moved from a
directory not yet listed into one already listed was seen under neither
name and its chunks were reported as orphans. The etcd store now
implements filer.EntryScanner: every page is read at the revision of the
first read, and a compacted revision fails the scan instead of falling
back to the walk.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* filer/etcd: read hard links at the scan revision
Hard-link records resolved during a scan were read at the current
revision, so a link updated meanwhile mixed two instants. Reads made with
the scan's context now use its revision.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* shell: list etcd among the stores volume.fsck reads from a snapshot
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* filer/etcd: skip hard-link records in a scan
Hard-link records live beside the entries under their random ids, so a
root scan could read one as an entry with a path made of random bytes,
which the traversal stream cannot send. Skip hard-link ids and keys whose
directory is not valid UTF-8.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* filer/etcd: recognize a hard-link record by the id it stores
A hard-link id can be valid UTF-8, so the key shape alone let some
records through a root scan as phantom files. The record's value is the
shared entry, whose HardLinkId is the record's own key; no entry key
can equal its own hard-link id.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* filer/etcd: scan entries under an invalid-UTF-8 directory
The filer sanitizes names but stores directories as given, so a path
written over HTTP can leave entries the listing never reaches. Skipping
them let volume.fsck count their chunks as orphans. Send them under the
sanitized directory instead, the substitution the filer uses for names.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* filer/etcd: fail a scan whose sanitized path names another entry
An entry under an invalid-UTF-8 directory is reported under the sanitized
directory. When that path is also another stored entry at the scan revision,
or another sanitized entry in the same scan, acting on the reported path
(volume.fsck deleting it) would reach the other file, so the scan fails and
names the stored path instead.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* ec: don't index ev.Shards[0] when the last shard was unmounted
* ec: skip a runtime emptied before the scrub merge reads its shards
FindAllEcVolumes releases each location lock before MergeEcRuntimes runs,
so an unmount can leave a merged runtime with no shards and the width
scan indexed Shards[-1].
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* ec: treat a scrub merge with no shard left as a vanished volume
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* ec: read Shards once in ShardSize so a concurrent unmount cannot empty it
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* ec: read each runtime's Shards once in the scrub merge
The vanished check and the slot construction read Shards separately, so
an unmount between them produced a non-nil view with no shards.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* ec: anchor the scrub merge on a generation that still holds shards
An unmount can empty the newest runtime while an older one keeps its
shards, which left a merged view anchored on the emptied generation with
no slots and the holder reported as skipped.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Mansur Akhmadi <10739460+mansur-akhmadi@users.noreply.github.com>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* shell: fs.mergeVolumes leaves in-progress multipart parts in place
CompleteMultipartUpload reads the part entries and later commits the
object with their fids, guarded only by the upload directory. A merge
that moved a part chunk in between deleted the source needle the new
object pointed at. Parts stay where they are until the upload completes
or is cleaned, and their chunks still count as referenced sources.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* shell: note in fs.mergeVolumes help that upload parts stay in place
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* shell: match upload parts when the buckets folder is the root
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* shell: count chunks behind a part's manifest as referenced sources
A skipped part whose chunks on a merge source sit behind a manifest was
not counted, so the merge printed no skip line and warned that nothing
references the source.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* shell: keep a part's direct source references when its manifest is unreadable
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* shell: count chunks resolved before an unreadable part manifest
ResolveChunkManifest returns the data chunks of earlier manifests along
with a later manifest's error; record them before returning so the
unreferenced-source warning does not fire for a source they reference.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* shell: test the partial manifest result without a timing delay
The unreadable manifest is now nested under a readable part manifest, so
the first level resolves fully before the failing lookup runs.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* filer_pb: add a depth-first traversal that can resume after a path
TraverseDfs walks a tree in listing order and reports each finished
path, so a caller can save one path and continue from it later. Pages
are read whole before visiting, and the entries of a page are visited
by up to five workers.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* filer.meta.backup: resume an interrupted metadata copy
The walk saves the last finished path every few seconds, and a walk
that continues an unfinished one starts after it. A fresh walk clears
the saved path first, and a finished walk clears it with the walk
start.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* filer.meta.backup: add -entriesPerSecond to throttle the metadata copy
Each entry the full copy reads waits on a rate limiter, and pages are
listed only after the previous page is copied, so the flag also bounds
how fast the copy lists the source filer. 0 keeps it unlimited.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* filer_pb: report each file of a depth-first walk once all before it are done
A throttled walk only reported progress after a whole listing page, so a
slow copy could go many minutes without a resumable path. Visit entries in
runs that end at a directory and report each file as soon as every entry
before it is done.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* volume: fail a vacuum step that finds a compaction already running
Compact, CompactByIndex and CommitCompact returned success when another
compaction held the volume. A second, uncoordinated vacuum (an expired
admin job still running, a shell volume.vacuum next to the worker) then
went on to commit or clean up the first run's .cpd/.cpx. Return a
"volume N is compacting" error instead, and refuse cleanup while a copy
is in flight, as the Rust volume server already did.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* volume: check the compaction files before a commit closes the volume
CommitCompact closed the needle map and data file and wrote the .cpc
marker before it found .cpd/.cpx missing. A duplicate commit, such as
the second of two overlapping vacuums, then left the volume registered
with no open files: it dropped out of the heartbeat and looked lost to
volume.list and volume.fix.replication until a restart. Check for the
files first so the refused commit leaves the volume as it was.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* volume: run makeupDiff before a commit closes the volume
CommitCompact closed the needle map and data file before makeupDiff, so
a makeupDiff failure (a compaction revision another commit moved, an I/O
error) returned with them closed: the volume stayed registered but
missing from the heartbeat until a restart. makeupDiff reads the live
files through its own handles while the commit holds the write lock, so
close them only once it has succeeded, as the Rust volume server does.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* volume: log a volume the heartbeat skips because its files are closed
A volume left without its needle map or data file vanished from the
heartbeat with no log line, so the master saw a replica disappear while
its files were still on disk. Warn on each skip, as the deleted-.dat
case next to it already does.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* volume: claim the compaction flag while cleanup removes the temp files
Cleanup only read the flag, so a compaction could start between the
check and the removal and lose its .cpd/.cpx. Claim it for the cleanup
instead, in Go and Rust.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* filer.sync: move the watermark over jobs that finished ahead of an older one
The watermark only took the finishing job's timestamp when it was the
oldest in flight, so jobs that completed before an older one were never
recorded and a quiet stream kept the offset behind finished events.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* filer.sync: keep one finished timestamp per gap behind a running job
A long job kept every later completion in the done heap. Only the newest
one between two running or failed timestamps can become the watermark.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* filer.sync: test done-heap compaction across several running and failed jobs
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* s3: answer GETs canceled before the first byte
A stream that fails before writing anything now always gets an S3 error
response, canceled errors included. The idle timeout and a shared lookup
canceled by another request both cancel the request context while the
client is still connected, and returning silently sent that client a
200 with Content-Length: 0. A client that has gone away never sees the
write.
Refs #12500
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3: mark inline GET write errors as written; bound the idle-cancel test
An inline object's write error comes after the status line, so the
caller must not write an S3 error on top of it. The idle-cancel test
caps the client wait and gives the response write its own deadline,
keeping only the read side on the 200 ms idle timeout.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3: test a GET whose request is canceled before the first byte
Drives GetObjectHandler through a chunk lookup that cancels the request
context, so a handler that returns before answering shows up as the empty
200 again.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3: check the cancel test reached the chunk lookup
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
* volume: forward no Content-MD5 for a JPEG the orientation fix rewrote
With -images.fix.orientation the primary stores and forwards the rotated
bytes, but sent replicas the client's digest of the original upload. The
replica checks Content-MD5 against what it receives and rejects the write,
so replicated JPEG uploads with a digest failed. The Rust volume server
already drops the digest once the body is rewritten.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* volume: test the forwarded Content-MD5 through PostHandler and a replica
The test now posts each upload to a primary whose fake master names a
real replica volume server, so it covers the JPEG gate, ReplicatedWrite
and the replica's own Content-MD5 check, and reads the stored bytes back.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* volume: fail the replica forwarding test instead of hanging when nothing is forwarded
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* volume: compare the primary and replica bytes in the replica forwarding test
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* filer.copy: compute chunk md5 so files get a real ETag
Chunks uploaded by filer.copy had no ETag, so every multi-chunk file got
the md5 of the empty string plus -N. An overwritten file kept its old ETag
and a revalidating client got 304 for changed content.
* filer.copy: store chunks in offset order
Chunks are collected in upload completion order, and the entry ETag
hashes the chunk ETags in stored order. Identical files copied with
concurrent chunk uploads got different ETags, and a file with swapped
equal-sized chunks could keep the old one.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* filer.copy: test chunk order and chunk md5 on chunked uploads
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* weed/shell: support -noLock flag in volumeServer.evacuate (#12501)
* shell: return flag parse errors from volumeServer.evacuate
A typo such as -noLock=invalid printed the parse error but the command
still succeeded, so a piped script treated the evacuation as done.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* shell: volumeServer.evacuate -h prints usage without failing
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* helm: set replication and disk on buckets created by the chart
* helm: test bucket replication and disk without ttl
* helm: reject unquoted bucket replication values
* helm: treat a null bucket replication as unset
A blank replication key in a values file failed the render with the
quoted-string error, while blank ttl and disk are skipped.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* helm: require a three-digit bucket replication
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* helm: note that removing bucket settings keeps the stored rule
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* helm: reject bucket replication above 255
fs.configure rejects an xyz value above 255, so the hook Job failed after
the release rendered.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* fix(shell): fs.log.purge purges every filer, not just one
Each filer keeps its own metadata logs in its own store, so purging a
single filer leaves the chunks referenced by the others and the volume
vacuum cannot reclaim anything.
The shell picked that single filer at random (commands.go:208) when no
-filer flag was given, and the admin script worker pinned it to
filers[0] (admin_script_handler.go:574), so the same command purged a
different filer on every run and never the whole cluster.
fs.log.purge now resolves the filer list from the master and purges each
of them, continuing when one is unreachable and reporting the failures
at the end. -filer=<addr>,... restricts the purge to a subset.
Also stops mutating the shared command environment per filer, so a
long-lived shell keeps its original filer.
* fix(shell): bound filer discovery and cover the cluster path
Address review feedback:
- resolveLogPurgeFilers waited on the master with context.Background(): a
shell whose master connection dropped would hang before purging
anything, where it used to purge the connected filer. Bound the master
round trip with a timeout and fall back to the connected filer when the
master cannot be reached or lists no filer.
- the tests only exercised the single-filer fallback. Rename that test to
what it actually covers and add one asserting every filer of the list is
attempted and the shared command environment keeps its original filer.
* shell: reject an fs.log.purge -filer list with no addresses
A comma or whitespace only -filer value fell through to master
discovery and purged every filer instead of the intended subset.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* shell: fall back to the shell's filer without a second master lookup
When the master listed no filer, FilerAddress() ran filer discovery again
and could wait out the shell's ConnectTimeout before failing. Also cover
master discovery with the fake discovery master.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* shell: test the fs.log.purge filer loop against a fake filer
The previous test looped over the addresses itself, so it passed no
matter what Do did.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Martin Morez <martin@morez.org>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* filer: validate bucket names before the SQL DDL hooks
The SQL store builds CREATE TABLE and DROP TABLE statements by formatting
the bucket name into the query. The DML paths already reject names that
fail VerifyS3BucketName, but the bucket-creation and bucket-deletion
hooks ran the DDL without that check, so a name reaching them through the
filer or WebDAV directory path (which skips the check for dot-prefixed
directories) could inject stacked statements. Apply the same validation
the DML paths use.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* filer: check that a valid bucket runs both DDL statements
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
* weed-db: end startup when password authentication fails
The failure branches returned the result of writing the error response,
which is nil on success, so the handshake carried on to ReadyForQuery.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* weed-db: test startup with wrong and unknown credentials
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* weed-db: check the auth error code and fail on stray read errors in the startup test
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* admin: escape file properties before rendering the modal
The Properties modal concatenated the entry name, path, MIME type and
extended attributes into HTML, so object metadata written over S3 could
inject markup into the admin page.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* admin: keep every property key when escaping
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* security: do not accept a path- or method-scoped filer token as admin
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* security: test that scoped filer tokens fail the admin token check
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* s3tables: let a table renamed across namespaces keep committing
* s3tables: keep the same-directory exception off a directory another table shares
A record written before metadata locations were confined to their namespace
(6f93f874) and claimed (#12473) can keep its files in another table's
directory, in another namespace. The exception that lets a table renamed
across namespaces keep committing to the directory it left behind accepted
such a record's commits too, where master refused them.
The namespace check now runs first, as on master. Only a location it refuses
is matched against the record's current directory, and the exception holds
only when no other record keeps its files there. Where other records' entries
sit is ignored, so a table created under the renamed table's old name, whose
files go to a directory of its own, does not take the exception away. The
scan runs only on that path, so commits inside the namespace cost what they
did.
Co-Authored-By: Claude <noreply@anthropic.com>
* s3tables: seed the table at the renamed table's old name
A rename may keep the old name taken, so creating a table there can be
refused; seeding the record keeps the test on the update path.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude <noreply@anthropic.com>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
filer: prefix bucket default collection, assign copy collection explicitly
S3 server-side copy paths (CopyObject/UploadPartCopy) assign volumes with an empty collection and relied on the filer's bucket fallback. The fallback used the raw bucket name, while PutObject uses getCollectionName(bucket) = <filerGroup>_<bucket>, so copies landed in a second, unprefixed collection.
Prefix the filer's bucket default with the filer group, and have the S3 copy path pass that collection explicitly, like PutObject. Direct filer writes keep path-rule precedence, so this only aligns S3 copy with S3 put.
Adds regression tests for the fallback, explicit collection precedence, and the S3 copy collection resolver.
* weed/s3api: avoid truncated marker rescan in ListObjects (#12540)
Signed-off-by: Tyagiquamar <mohdquamartyagi@gmail.com>
* fix(s3): validate marker directory candidates
Respect prefix filters and internal-directory exclusions when recovering marker candidates. Restrict lookups to direct children, propagate read failures other than not-found, and deduplicate directory marker echoes.
Add regression coverage and model promoted objects consistently in the XML-unsafe prefix fixture.
---------
Signed-off-by: Tyagiquamar <mohdquamartyagi@gmail.com>
Co-authored-by: Chris Lu <chris.lu@gmail.com>
* s3vectors: bound body buffering before authentication
* docs: include vector engine installation and source builds
* s3vectors: prevent MIME sniffing of JSON error responses
* docs: load vector Cargo configuration when building from source
* docs: note the native protoc requirement for Windows vector builds
* docs: make vector source checkout and working directory explicit
* ci: handle deb822 sources during Ubuntu mirror failover
* ci: bound Ubuntu mirror attempts and reject partial index updates
* vector: expose request metrics and registration-aware health endpoints
* vector: report immutable read cache and storage fetch metrics
* docs: describe vector engine health, metrics, and maintenance
* vector: count canceled requests when their handler is dropped
* vector: account for uncached large immutable storage reads
* vector: reliably release the stalled credential reader in tests
* vector: observe large reads as their streamed body is consumed
* vector: bound heartbeats on established master registration streams
* test: exercise hardened vector maintenance on Go and Rust volumes
* vector: register engines in their configured filer group
* vector: make retried index creation preserve compatible datasets
* vector: allow registration to advertise a separate address
* test: release stalled credential readers without blocking FIFO cleanup
* test: allow hardened vector coverage in a named filer group
* vector: discover engines in the same filer group as their gateway
* vector: fall back when the advertised address is empty
* admin: expose vector readiness, maintenance, and management tools (#12520)
* s3vectors: share vector discovery with trusted management clients
* admin: show vector engine readiness and recent maintenance activity
* admin: suggest S3 Vectors IAM actions from the supported operations
* admin: manage vector indexes, policies, tags, encryption, and query data
* admin: validate SSE-S3 support before creating encrypted vector buckets
* admin: discover vector engines in their registration group
* admin: probe vector engines concurrently with independent time budgets
* admin: verify encryption capability on the remote S3 gateway
* admin: defer discovery until after the HTTP server can start
* admin: distinguish unknown vector health services from failed engines
* admin: count workers that can execute vector index maintenance
* admin: allow read-only users to query vectors and inspect tags
* vector: scope gateway and admin discovery to the configured filer group
* admin: refresh policy state after mutations and serialize policy controls
* vector: drain active RPCs before closing replaced discovery connections
* vector: respect request cancellation while waiting for discovery
* vector: prefer another registered engine after a transport failure
* admin: check all masters before reporting no registered vector engines
* iam: honor attached vector policies without broadening legacy grants
* vector: allow short-lived discovery clients to close their connections
* vector: refresh cached master addresses for each discovery lookup
* iam: enforce vector bucket tag conditions in attached policies
* iam: authorize vector bucket creation with the requested tags
* test: exercise vector IAM grants and tag denies on Go and Rust volumes
* admin: reject authenticated vector mutations without an admin role
* test: run vector IAM assertions before the shared engine is stopped
* vector: add native mini startup, shell tools, and a runnable Compose example (#12522)
* s3vectors: share vector discovery with trusted management clients
* admin: show vector engine readiness and recent maintenance activity
* admin: suggest S3 Vectors IAM actions from the supported operations
* admin: manage vector indexes, policies, tags, encryption, and query data
* admin: validate SSE-S3 support before creating encrypted vector buckets
* admin: discover vector engines in their registration group
* admin: probe vector engines concurrently with independent time budgets
* admin: verify encryption capability on the remote S3 gateway
* admin: defer discovery until after the HTTP server can start
* admin: distinguish unknown vector health services from failed engines
* admin: count workers that can execute vector index maintenance
* admin: allow read-only users to query vectors and inspect tags
* vector: scope gateway and admin discovery to the configured filer group
* admin: refresh policy state after mutations and serialize policy controls
* vector: drain active RPCs before closing replaced discovery connections
* vector: respect request cancellation while waiting for discovery
* vector: prefer another registered engine after a transport failure
* admin: check all masters before reporting no registered vector engines
* iam: honor attached vector policies without broadening legacy grants
* vector: allow short-lived discovery clients to close their connections
* vector: refresh cached master addresses for each discovery lookup
* iam: enforce vector bucket tag conditions in attached policies
* iam: authorize vector bucket creation with the requested tags
* test: exercise vector IAM grants and tag denies on Go and Rust volumes
* admin: reject authenticated vector mutations without an admin role
* test: run vector IAM assertions before the shared engine is stopped
* mini: optionally start and supervise the vector engine
* mini: bootstrap vector buckets with existing Lance catalog helpers
* shell: manage vector buckets, indexes, and vector data
* docs: provide a runnable vector and maintenance worker Compose stack
* shell: scope vector discovery to the session and close its connections
* mini: honor the bind address for vector and metrics listeners
* mini: reserve vector ports before assigning gRPC and compatibility listeners
* mini: wait for vector storage setup and master registration
* mini: create requested vector buckets without requiring the Lance HTTP listener
* shell: refresh master addresses during vector discovery
* mini: resolve sibling vector executables with platform extensions
* vector: preserve uncertain creates and fix mini listener allocation (#12525)
* vector: retain catalog records when index creation has an unknown outcome
* mini: keep chosen listener ports reserved throughout allocation
* mini: resolve hostname binds before starting vector listeners
* vector: fix integration test imports and recover interrupted index creation (#12533)
* shell: register vector operations from the command package
* s3vectors: resume interrupted index initialization safely
* mini: exclude disabled WebDAV from listener allocation
* s3vectors: retain uncertain creates across refused retries
* s3vectors: claim pending indexes before retrying initialization
* s3vectors: compare resolved distance metrics during recovery
* s3vectors: preserve metadata keys when retrying initialization
* s3vectors: verify ownership before initializing new catalog records
* helm: deploy vector engines with TLS, probes, and monitoring
* helm: provision vector buckets through the S3 Vectors API
* helm: bracket IPv6 pod addresses for vector listeners
* helm: discover vector engines through the all-in-one master
* helm: accept vector component NetworkPolicy overrides
* helm: bind vector metrics to the pod address on both IP families
* helm: provision vector buckets through embedded and all-in-one gateways
* docs: use the full Helm name for external vector certificates
* helm: render vector instance labels as strings
* helm: allow vector bucket hooks to use restricted security contexts
* helm: honor vector security context enable switches
* helm: prevent duplicate vector monitoring labels
* helm: reject metrics flags with the default pre-metrics vector image
* test: skip vector chart compatibility checks when Helm is unavailable
* test: pass installer scripts through stdin on Windows
* ci: allow cold Intel macOS vector builds to finish
* ci: retain completed Rust builds when later platform checks fail
* ci: build, test, and publish vector engines for macOS and Windows
* installer: install native vector archives on macOS and Windows
* ci: run desktop vector checks for shared Rust and protobuf changes
* ci: publish vector releases from a separate job with write permissions
* ci: build tagged vector releases without restoring caches
* ci: propagate Windows bash path conversion failures
* grafana: add vector API, engine, cache, and maintenance panels
* helm: include vector panels with Kubernetes metric selectors
* grafana: preserve cluster and namespace identity when matching engine health
* grafana: retain engine identity when scrapes fail
* grafana: show zero cache hits before the first hit
* grafana: keep cluster and namespace identity in vector cache ratios
* topology: list collections without nesting the collection map read lock
Items() already takes the read lock and copies the values. Holding it
around the call made ListCollections take the read lock twice, so a
DeleteCollection waiting for the write lock in between blocked the
second read lock and wedged the master.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* topology: test ListCollections against a concurrent collection delete
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* master: move the KeepConnected stream cleanup into removeClusterNode
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* master: drop a filer from the lock ring only with its last registration
A filer's replacement can register under the same address before the
master ends the old stream. The cluster list counts both registrations,
but the old stream's cleanup removed the address from the lock ring
outright, leaving the live filer out of the ring until it re-registered.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* master: order a stream's cluster list and lock ring changes
The cleanup of an ending stream could see its address removed from the
cluster list, then a new registration of that address add it to the list
and the ring, and only then remove it from the ring.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* master: take a leaving filer's reconnect out of the lock ring
A filer that reconnects while leaving registers with LeaveLockRing set.
Its broken stream can still be open on the master, and since that
stream's end no longer drops an address that is still registered, the
reconnect removes the address from the ring itself.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* helm: say what the Lance worker needs from security.toml
Since #12470 the Lance namespace authenticates weed-worker by a token
derived from jwt.filer_signing.key, which the chart renders only with
securityConfig.jwtSigning.filerWrite. Without it, a gateway with
identities lists the worker no tables, and the lance_* jobs silently
find nothing to do.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* helm: warn that filerWrite also requires signed filer HTTP writes
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* s3: encrypt volume data when the filer is configured to
weed s3 read the filer's configuration at startup (buckets dir, filer
group, masters, metrics, maxMB) but not its -encryptVolumeData, so a
cluster that turned encryption on at the filer still had S3 uploads
stored plain, while uploads through the filer itself were encrypted.
The gateway now encrypts when either its own -encryptVolumeData or the
filer's is set; weed-vector, weed-worker and weed-vector-cli already
follow the filer's.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* vector: weed-vector, weed-worker and the CLI take volume encryption from the filer
They already encrypted chunks for the volume servers when the filer's
configuration said so, or when their own --encryptVolumeData did. With
weed s3 following the filer too, the filer's -encryptVolumeData is the
one switch for every writer, so the three flags are removed.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* S3 Vectors tests: turn volume encryption on at the filer
VECTOR_ENCRYPT_VOLUMES=1 set -encryptVolumeData on the gateway, the
engine and the worker. It now sets it on the filer alone, so the leg
checks that each of them follows the filer: the gateway's stored chunks
carry a cipher key (with a gateway that ignores the filer, the leg fails
on "the gateway stored a chunk without a cipher key"), and so do the
engine's.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* weed-worker README: volume encryption follows the filer
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* s3: help text says the filer's -encryptVolumeData turns S3 encryption on
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* S3 Vectors tests: plain legs check the engine stored no cipher key
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
vector: the engine's, CLI's and worker's help say what they do now
weed-vector's --masters and --filer-group said "reads through" and
--security-toml named only gRPC TLS, though the engine keeps tables on the
filers and volume servers and takes https.client, the read JWT key and the
SSE-S3 keys from it too; weed-vector-cli called itself a "query spike";
weed-worker's --filer-signing-key spoke of write credentials the
namespace no longer vends.
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
vector: weed-vector drops the ignored --namespace and --filer-signing-key
Since #12468 the gateway resolves every table it sends, so the engine has
no catalog to name or authenticate to; the two flags were kept only so an
older command line still started, with a warning. The direct-storage
engine is new, and release 4.52's command lines need changing anyway
(--storage-option is refused), so they are removed.
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
vector: refuse an SSE-S3 index when the engine has no key for it
The engine writes and reads an SSE-S3 bucket's files itself, so without
s3.sse.kek or s3.sse.key it failed inside Lance and the client saw only
InternalFailure. weed mini makes this easy to hit: it generates its key
for its own S3 server only. A table the gateway marks sse_s3 is now
refused up front with FailedPrecondition, which the gateway answers as
ValidationException with the reason.
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* s3: don't report a version delete as done when the filer lookup failed
deleteSpecificObjectVersion treated any getEntry error as "entry is gone":
- the null-version precheck returned success without deleting anything,
- the recheck after a failed rm returned success while the entry may
still be there,
- a failed .versions lookup was read as "not the latest version", so the
latest blob was removed without repointing the pointer first.
Only filer_pb.ErrNotFound now counts as a miss. Other errors fail the
delete, so the S3 client or the lifecycle worker retries it. The lookups
go through retryFilerOp, like the other .versions reads in this file, so
a single transient filer error doesn't turn into a 500.
* s3: include recheck error when a version delete fails
* s3: recheck a failed version delete once instead of retrying
---------
Co-authored-by: Mansur Akhmadi <10739460+mansur-akhmadi@users.noreply.github.com>
* volume.fsck: batch and deduplicate filer entry deletions
In volume.fsck with -findMissingChunksInFiler and -reallyDeleteFilerEntries, deleting orphaned filer entries was slow and CPU intensive due to tiny worker channel buffers (128), lack of batch processing, redundant HTTP DELETE requests for multi-chunk files, and lock contention on parent directory tracking.
Optimize by:
- Adding startPurgeBatchWorkers with 4096-buffered channels to prevent producer stalls and batch opportunistic deletes up to 64 items.
- Deduplicating missing chunks per file path during volume checks so that multi-chunk files are queued once.
- Processing batches with in-batch path deduplication and aggregating parent directories before updating purgedDirs in a single lock acquisition.
- Buffering temporary fid file writes (64KB) during the traversal to avoid millions of unbuffered syscalls.
* volume.fsck: report a failed write of the filer fid files
The fid files are written through a buffer now, so the last part of
each one reaches the disk only in the deferred Flush. Its error was
dropped: a full temp disk left a short fid file and the collection
still succeeded, and the volume check would then take the chunks
missing from it for orphans.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* volume.fsck: retry a batched path whose earlier delete failed
A path was marked done before its delete ran, so a second copy of it in
the same batch was skipped after a transient failure and the entry
stayed. Only skip paths already deleted.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* volume.fsck: keep reporting a failed DELETE response read
The non-verbose path now discards the body without checking the copy,
so a connection cut after the headers went unreported.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* volume.fsck: keep one purge worker starter and one delete function
startPurgeWorkers and the single-path httpDelete were left only as
wrappers the tests called. Give their names to the batch versions and
adapt the tests.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* s3tables: give a table created without a location its own directory
An Iceberg table created without a location kept its files under its name
path, so a rename left them where the next table of that name would land.
Such a table now gets s3://<bucket>/<namespace>/<name>-<uuid>, stored as the
record's warehouseLocation and returned by GetTable; its first commit writes
there without moving the root. Records without one keep resolving to their
name path, so existing tables need no rewrite. Lance datasets stay at their
name path, where Lance clients and the vector engine look for them.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* iceberg: create tables and views in a directory of their own
A table or view created without a location took its name path, or a
unique one only when that path was already taken. It now always gets
<name>-<uuid>, so a later rename never leaves its files under a name
another table can take.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* iceberg: load and commit a table at its warehouse location
A table created through S3 Tables without a location has no metadata until
its first commit; loading or committing it now uses the directory recorded
for it, and only a record without one falls back to its name path.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* iceberg: fail a stage-create whose marker is not saved
The marker is how the commit finds the location the stage-create handed
out. Without it the commit fell back to the name path while the client's
files sat under the returned location.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* test: expect tables created without a location in their own directory
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* test: rename a table that keeps its files elsewhere before reusing its name
Enterprise keeps the storage marker on a renamed table's files, so the name
of a table whose files sit under its name path stays taken there.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3tables: remove the table directory when a create backs out
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* iceberg: answer a create that lost the race with the registered table
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* iceberg: answer a view create that lost the race with the registered view
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* iceberg: fail a staged commit whose name another create registered
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* iceberg: remove the directory of a create whose name another table took
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* iceberg: keep the metadata of a staged commit whose name is taken
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* test: write the lakekeeper table's data under its warehouse location
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* iceberg: commit a staged table at the location of its own stage
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* iceberg: never continue another stage when a commit's own marker is gone
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* iceberg: test filer honors exclusive creates and token-guarded deletes
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* ec_balance: make the per-rack cap on global moves a worker setting
The global load-balancing phase planned at most 10 moves per rack per
detection run, hard-coded. On a cluster where one disk per machine holds
several times its siblings' shards, that is about 20 moves an hour against
tens of thousands needed, however many moves the worker can execute.
Add global_max_moves_per_rack to the plugin worker config. The default stays
10, so nothing changes unless it is raised. It is clamped to 1..5000 because
the planner reads 0 as unlimited and would plan a whole drain in one run.
Claude-Session: https://claude.ai/code/session_011UpvafcfUuC2mLxNAXLQJi
* ec_balance: clamp global_max_moves_per_rack before narrowing it to int
A value past the int32 range made ReadIntConfig return the default, so a
request asking for more than the maximum got 10 moves instead of 5000.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* ec_balance: test the configured global cap through Detection
The helper test still passed if Detection handed the planner a literal cap.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* ec_balance: plan no more global moves per rack than the detection keeps
Each global move scans the source node's volumes, and the detection keeps
at most max_results moves. With the cap at 5000, three skewed racks of
20,000 volumes planned 15,000 moves to keep 500, taking about 70 s of the
300 s detection budget; bounded by max_results it takes about 5 s.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* ec_balance: test the configured global cap with a result limit
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* filer.remote.sync: -allMounts writes back every mounted directory from one process
A filer with many remote mounts needs one filer.remote.sync process per
mount. -allMounts runs the existing -dir write-back once per directory in
the mount table, in one process: each with its own subscription, client id
(the filer keeps one subscription per client id), offset, processor and
retry loop. A supervisor reads the mount table every -mountsCheckInterval
(5s) and starts a write-back for each mounted directory that has none
running, 10 at a time, 500 ms apart. It never stops one because the live
table changed: a write-back ends itself where the -dir process exits, at the
first mount-table version it receives that ends its mount (#12472), and a
directory still mounted is started again from its saved offset, after a
pause that grows from 1s to 30s while it keeps ending within a minute. A
directory mounted while the process runs, without a saved offset, waits up
to 2 minutes for one (a mount followed by a pull sets it after the pull). A
write-back that ends at an unmount 5 times in a row while the table still
mounts it at the same location is logged as a warning.
To run under a supervisor, a follow:
- returns errMountGone at the end of its mount instead of exiting.
admitUntilMountEnds takes the end action: the -dir process keeps draining
and exiting as before; a -allMounts follow admits nothing more (events
after the end can still arrive in the same batch or before the stream
closes, including remote.unmount's purge and a remount), ends its stream,
drains, and saves no offset.
- drains its admitted jobs before it returns, however it ended, so a
successor never races a job still running for the same path.
- saves its offset only over the value it read at its start or last saved
(offsetGuard), both at its checkpoints and at its end. Otherwise a
write-back behind in the log can overwrite the offset remote.unmount
writes after its purge (or a mount after its mapping write), and its
successor replays the unmount and never writes the remounted directory
back. Once it sees the offset moved it saves nothing more and keeps no
resume position. A value whose write reported an error counts as its own
when read back.
- reads the stored offset once, before the mount table, and uses it as both
its start and its guard's value: read after the table, an unmount
completing in between would start a follow past the unmount's table.
- keeps a resume position for its next attempt only with the mount location
it was earned under and the offset stored at the time; it is dropped when
the directory is mounted elsewhere or someone else moved the offset.
- turns a panic in a job into that job's error, so one directory's failure
does not end every directory's write-back.
Fixes#12471
* filer.remote.sync -allMounts: retire a failed follow whose mount is gone
A follow that failed before its stream reached the unmount (or before it
started) was retried every 30s forever: DetectMountInfo reports the
directory not mounted, which is not errMountGone. No stream runs on the
retry path, so a directory the live table no longer maps is retired there
and started again by reconcile if it is mounted again.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* filer.remote.sync -allMounts: drop the restart pause of an unmounted directory
ended() left notBefore and ends behind for every directory ever mounted.
reconcile now drops them once the directory is neither mounted nor
followed.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* filer.remote.sync: log a -dir drain held at the mount end
The -dir exit drained with processor.Drain, so a remote call hung during
an unmount or move held the process silently; the follow's own drain
already warns every 5 minutes.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* filer.remote.sync: keep the resume position through a failed mount read
A transient DetectMountInfo error dropped the only position behind the
event a follow failed on. With -timeAgo and no checkpoint saved yet, the
retry then started at a later now minus timeAgo and skipped that event.
The next attempt that reads the table still checks the position against
the mount location and the stored offset.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* filer.remote.sync: judge a kept resume position by the directory's remote location
resumeFor compared the mount's raw location, so a -dir below a mount kept
its position when the same remote was mounted one level up (or down),
although its paths now map to other keys; a replay from that position
skips startAfterMountOffset and writes old events to the new keys. The
position now carries dirRemoteLocation and is checked against it.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* filer.remote.sync -allMounts: warn about a replayed unmount only from an unmoved offset
The warning counted every end at an unmount while the directory stayed
mounted at one location, so write-backs that each saved progress, or
unmount and remount cycles between two reconciles, also tripped it, and
it told the operator to set the offset to now, skipping events. An end
saves no offset, so a run of ends is one replay only while the stored
offset stays put: a moved offset now starts a new run.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* filer.remote.sync -allMounts tests: wait for the supervisor before moving the frozen clock
A reconcile or end recorded after the clock moved would start its wait
or pause at the moved time, which the frozen clock never passes.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* remote.mount: save the sync offset right after the mount mapping
A new mount had no saved offset, so filer.remote.sync -allMounts waited
the full allMountsOffsetWait before following it and then started from
the directory's creation time anyway. remote.mount and remote.mount.buckets
now save the time taken just before the mapping write, keeping an offset
remote.unmount saved for a remount.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* remote.mount: save the directory's creation time as the sync offset
The mount time skipped local files already in a -nonempty directory, which
a write-back with no offset replays from the directory's creation. Saving
that creation time keeps the start position and only ends the -allMounts
wait.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* remote.mount: warn instead of failing when the sync offset is not saved
The mount mapping is already saved by then, and a mount without an
offset still starts from the same creation time after the -allMounts
wait. Failing stopped remote.mount.buckets before the remaining buckets.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* ecbalancer: drain a full server through any target that can take a shard
The global phase paired the fullest server with the single emptiest one and
stopped for the whole rack when that pair had no legal move. Where every
volume already has a shard on each machine, no cross-machine move keeps the
spread, so on hosts running one volume server per disk the first server's
disk kept its pile while its siblings on the same machine stayed nearly
empty: the phase never planned a move.
Try sources fullest first and targets emptiest first and take the first pair
with a legal move, so such a server drains onto its own machine's emptier
servers. A same-machine move goes only to a server holding none of the
volume, so one disk never carries two of its shards. Per-machine counts are
indexed once per rack and pairs without a move are remembered until a move
can change them, so an uncapped drain stays fast.
Claude-Session: https://claude.ai/code/session_011UpvafcfUuC2mLxNAXLQJi
* ec_balance: keep global moves ahead of within-rack moves under the detection cap
Detection truncated the plan to the per-cycle job limit in planner order,
which puts the global moves last. A long within-rack backlog pushed all of
them past the limit, so even a planned global move never ran. Order dedup
and cross-rack first, then the capped global phase, then within-rack.
Claude-Session: https://claude.ai/code/session_011UpvafcfUuC2mLxNAXLQJi
* ec_balance: keep each volume's moves in plan order when promoting global moves
The admin scheduler runs only the first proposal per volume each cycle. A
global move promoted ahead of its volume's within-rack moves can take a shard
that one of those moves placed, so it ran while the shard was still on the
original server and failed.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* ecbalancer: order global targets by fullness after taking the shard
The search stops at the first target that would end up fuller than the
source, which only holds when targets are ordered by that same value. With
unequal capacities an empty one-slot server sorted first and ended the search
before a larger server that could take the shard.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* ecbalancer: free the source disk slot on a global move
The global phase cleared the moved shard's bits but left the source disk's
free slots and shard count unchanged, unlike every other phase. Use
releaseShard so a later pick of that server sees the freed disk.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* ec_balance: test that a capped detection keeps a global move
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* ecbalancer: test the same-machine move skips a sibling holding the volume
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* iceberg: keep a losing table create from deleting the winner's metadata
* iceberg: keep a staged file whose published record was renamed away
A create answers a conflict when its record was already visible and
another request changed it before the create could back it out. The
cleanup then looked only at the original name, so a record renamed in
between read as absent and the file it still points at was deleted.
S3 Tables now reports that case with a conflict of its own, and both
create paths keep the staged file on it instead of probing the name.
A conflict over a still-hidden record, lost to another create, still
removes the file.
* iceberg: discard a losing create's metadata only while it is still its own
The cleanup deleted the staged file by name. The path can be reused
before it runs: the table is dropped and purged, and another create
stages the same v1.metadata.json and registers it, which the losing
request then deleted.
Staged files now carry a per-request token in an internal extended
attribute, and the cleanup deletes with that token as the filer's
ExpectedExtended precondition, checked under the path lock the delete
runs under. A file that took the path since is left in place.
* iceberg: classify a typed S3 Tables error by its code, not its message
isS3TablesAlreadyExists matched "already exists" in the message before
looking at the code, so an InternalError whose text mentioned an
existing entry read as the name being taken: CreateTable answered 409
instead of 500, and registrationRefused discarded a staged file the
catalog may still point at. A typed error now decides by its code; only
an untyped one is matched by message.
* iceberg: pin stage-create on in the stage-create conflict test
Stage-create is on by default, but a shell that sets
ICEBERG_ENABLE_STAGE_CREATE=false turned the expected 409 into a 501.
Setting it in the test keeps the result independent of the environment.
* iceberg: test two creates overlapping at the canonical metadata name
The concurrent-creators test only lines the creators up before they
start, so whether two of them ever collide at v1.metadata.json is up to
the scheduler. The new test fixes the order: the second create runs
while the first has staged v1.metadata.json and not yet registered it.
It checks that every staging attempt is exclusive, that the second falls
back to v1-<uuid>, that both files hold their own metadata while both
are staged, and that the loser removes only its own file.
* s3tables: a create whose unmark write never went out stayed hidden
confirmChild counted any failure to take a create's mark off as possibly
published, the lookup that precedes the write included. That lookup
failing leaves the mark on, so the record stayed hidden; if another
create then adopted it, the back-out reported a published record that
changed, and the Iceberg catalog kept a staged file no record names.
The mark-removal calls now go through a wrapper that notes whether an
UpdateEntry went out without the filer refusing it. Only then may the
record have been seen. A refused compare, a missing entry, or a failure
before any write leaves the create's record hidden.
* s3tables: pass mayBeVisible into backOutChild
Fixes the build after rebasing onto the backOutChild split.
* volume: check a needle's CRC the first time a range of it is read
A ranged read never checked the CRC, so a replica with a damaged needle
served the damaged bytes with 206 and the reader never tried another
copy. The first ranged read of a needle now streams the whole needle
through the existing CRC check, with the usual pooled page buffers,
before writing any byte. A mismatch fails the read, so the reader falls
back to another replica. Needles that pass go into a bounded LRU, keyed
by volume, compaction revision, id, append time and checksum, so later
ranges of the same needle stream as before.
Fixes#5959 for ranged reads.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* volume: key verified ranges by volume load, share concurrent checks
From review: the cache key read v.SuperBlock.CompactionRevision without
the data file lock, racing a compaction commit that reloads the super
block, and a remounted copy of a volume inherited the old copy's
verified entries. Key entries by a load generation instead, an atomic
counter that changes on every load, including the reload in
CommitCompact, and cache a result only if no reload happened during
the check. Concurrent first ranges of a needle now share one check
through singleflight instead of each reading the whole needle.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* volume: keep hot needles verified, leave the io error streak to the range read
From review: Contains does not refresh an entry's place in the LRU, so
even a constantly read needle was evicted after 64k other needles were
verified; look it up with Get instead. The verification pass also
cleared the io error streak before the range read ran, so a range read
failing with EIO after a passing check never built up a streak; the
pass into io.Discard now leaves the streak to the range read, like the
meta read before a streamed GET.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* volume (Rust): check a needle's CRC the first time a range of it is read
Mirror of the Go change: a range served from the data file now checks the
whole needle's CRC before any byte of it is sent, so a damaged replica
answers 500 and the reader tries another copy. Needles that pass go into a
bounded LRU keyed by the volume's load generation, needle id, append time
and checksum; concurrent first ranges of a needle share one check, and a
check that passes leaves the io error streak to the range read.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* volume (Rust): a range over the whole needle checks its CRC
Go reads a range that covers the whole needle through the checked full-read
path. The Rust volume server read it from the data file unchecked, so
"bytes=0-" served a damaged needle with 206. Check the slice's CRC before
answering, as the full read does.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* volume: check the covered blocks of every range against CRCs recorded at the first check
The first ranged read of a needle now records the CRC of each block
(64 KiB to 1 MiB, at most 64 blocks below 64 MiB) while it checks the
whole needle. Later ranges read the block-aligned span around them and
write a block's bytes only once it matches, so damage that appears
after the first check is not served.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* volume (Rust): check the covered blocks of every range against CRCs recorded at the first check
verify_checksum now records a CRC per block (same sizes as Go's
rangeCheckBlockSize) and returns them from the cache; read_checked_range
reads the block-aligned span of a range and answers 500 if a block no
longer matches, so damage after the first check is not served.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
* sw-client: the filer and volume client library
The client the Enterprise repository's VFS and vector engine use, with
the same API: filer gRPC for entries, listings, volume locations,
assignments and remote caching; volume HTTP, plain or TLS, for needle
bytes; chunk resolution for sparse, overlapping and manifest chunks; a
location cache; and the write side. Enterprise-only protocol fields
(license, conditional mutations, hard links, KV listing) are left out.
Compiled against weed/pb/filer.proto like the other worker crates.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* weed-lance-worker: reads straight from the filers and volume servers
direct_read::DirectReads wraps the object store Lance opens so that
entries, listings and bytes come from the filers (gRPC) and the volume
servers (HTTP, signed with the read JWT), whatever the entry holds:
inline content, chunk manifests, sparse or overlapping chunks, the
filer's volume encryption, the gateway's SSE-S3 (sse.rs, with the KEK
from security.toml or the filer) and objects tiered to remote storage.
Writes still go to the inner store. The code and its tests are those of
the Enterprise repository's engine.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* weed-vector: read tables from the filers and volume servers
The engine asks the masters for the filers (and follows the list), takes
jwt.signing.read.key, grpc.client, https.client and the SSE-S3 keys from
security.toml, and opens every table with DirectReads innermost and the
read cache outside it. --masters is required: the gateway finds the
engine through them and the engine finds the filers. Writes still go to
the gateway.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* sse: chunks out of a manifest keep their own SSE-S3 keys
A chunk expanded from a manifest carries its id as fid, so keying the
per-chunk key cache by file_id gave every such chunk the first one's key
and IV. Keyed by the chunk's id however it is carried.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* direct reads: a silent filer is passed over like an unreachable one
Each filer attempt is bounded, so a filer that accepts the connection and
never answers does not hold the read or the startup lookup.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* sw-client: carry the assignment's fsync to the volume upload
The filer's storage rule may want a needle on disk before the volume
answers; the assignment says so and the upload now asks for it, as the
Go uploader does.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* weed-vector: argument tests pass the required masters; connect timeout for filers
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* direct reads: a failed read is tried once more against fresh locations
A fetch that fails forgets the volume locations it was made against; the
request now looks them up again and retries instead of failing on a
volume that moved.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* sw-client: an exclusive create that finds its entry, and needle deletion
create_entry marks ENTRY_ALREADY_EXISTS with EntryExists, so a caller can
tell a lost conditional create from a failure; the transport deletes a
needle whose entry was never created.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* sse: encode chunks as the gateway stores them
EntryEncoder writes SSE-S3 under a fresh object key wrapped by the KEK,
serialized with the IV, key id and commitment the gateway's reader
expects, each chunk under the IV at its offset; and the filer's volume
encryption under a per-chunk key.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* direct storage: writes, deletes and table removal without the gateway
A put uploads its chunks to the volume servers the filer assigns, at
the filer's chunk size, encoded as the bucket and cluster ask, and
creates the entry the gateway would: md5, mime, size, the ETag
attribute, the SSE-S3 attributes. A conditional create is the filer's
exclusive create, so Lance's manifest commit conflicts the same way;
the needles of a create that failed are discarded. A multipart upload
turns parts into chunks and derives its ETag from them. Deletes remove
the entry with its data, and remove_table a table directory. Copy and
rename, which Lance does not use on S3, are not offered.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* vector.proto: Table says whether its bucket encrypts with SSE-S3
The engine writes the table's files itself, so it needs what the S3 write
path knew: whether the bucket declares SSE-S3. storage_options go: the
engine reaches no S3 store.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* s3vectors: send the engine the bucket's SSE-S3 setting
The rule the S3 write path applies to a Lance bucket's files, factored out
of DeclaredBucketEncryption, decides it from the vector bucket's record.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* weed-vector: every table operation goes to the filers and volume servers
Opens, index creation and drops all go through the direct store, so the
engine has no S3 client of its own left to protect: --storage-option,
the endpoint pin and the request-option allowlist go, and lance's own S3
store is built unsigned and never dialed. The bucket's SSE-S3 setting
comes with the table; -encryptVolumeData encrypts chunks for the volume
servers as the gateway's flag does.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* s3vectors tests: the engine takes no credentials
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* s3vectors e2e: the engine writes what the gateway would have
Every leg checks that the entries the engine wrote carry the gateway's
ETag attribute, the hardened leg that their chunks are encrypted for the
volume servers, and a new test that an SSE-S3 bucket's files are written
under keys the gateway unwraps when it serves them over S3.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* cluster plumbing moves to the worker crate for the engine, worker and CLI
security.toml, the gRPC TLS connector and acceptor, the master dialer,
the filer discovery and the storage connection built from them now live
in weed-lance-worker (cluster, security, grpc_tls), where the worker and
the CLI can use them too. The server crate re-exports security and tls
under their old paths.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* weed-worker, weed-vector-cli: tables through the filers and volume servers
dataset::Storage replaces the S3 credentials every job and the CLI
carried: a table is opened and written through the cluster's storage,
encrypted as its bucket declares (read from the bucket's record on the
filer), and dropped there. Both take --masters and security.toml like the
engine; --access-key and --secret-key go, and with them the catalog's
x-api-key credential, which the cluster token replaces.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* lance catalog: the cluster token alone authenticates a cluster process
The engine, the maintenance worker and the CLI hold the cluster's filer
signing key and reach a table's files themselves. A request presenting
the cluster token and no other credential is theirs: it is authenticated
by the token, runs through the trusted manager, and is vended no S3
credentials.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* worker chart, README and test design follow the worker's new arguments
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* s3vectors tests: the worker and the CLI reach the cluster through the masters
The cluster gets a filer signing key, which the catalog authenticates
the worker and the CLI by; both take --masters in place of S3 keys.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* direct storage: encrypted chunks each start their own CTR stream
The gateway encrypts a multipart part from the IV derived for its offset,
from position zero; the encoder encrypted from the exact byte offset of
the object's stream instead, which agreed only at block boundaries. A
part of 3,000 bytes read back wrong; now tested.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* direct storage: keep what a lost reply saved, discard what a failed put left
A create the filer saved but did not answer, or an exclusive create
retried onto this very upload, keeps its chunks and counts as written;
the needles of a put that failed midway are discarded. The report prints
reads as reads.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* weed-worker: the security.toml it is given also holds its signing key
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* worker chart: the Lance worker mounts grpc.client's certificate
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* direct reads: the filer deadline bounds a listing page, not the listing
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* weed-vector: security.toml booleans read as viper reads them
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* weed-vector: a filer the master lists badly does not hide the others
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* weed-vector: register with the masters only once ready to serve
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* sse: every chunk after the first is written under a key of its own
Chunks of one object shared a key with IVs derived from their offsets, so
two chunks meeting inside a cipher block shared keystream. The first
chunk stays under the object's key and IV, which is how the gateway reads
an object of one chunk; every other chunk gets a fresh key and IV, as the
gateway's multipart parts have and as it reads an object of several.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* worker chart: the Lance worker is given the filer group
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* s3vectors e2e: the gateway serves engine-written SSE-S3 files of one chunk and of several
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* sw-client: the HTTP transport asks the volume to fsync when the assignment does
write_synced fell back to the trait default, which refuses fsync, so every
upload under a fsync storage rule failed before it was sent.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* sw-client: https.client.ca alone vouches for volume servers
With a CA configured, reqwest still trusted the system roots next to it;
weed processes trust only the configured pool.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* sw-client: an empty volume range is read without a request
A zero length underflowed the Range end, and a range past u64::MAX
overflowed it; the first now returns no bytes, the second an error.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* sw-client: a whole-needle answer to a ranged read is cut to the range
A server or proxy that ignores Range answers 200 with the whole needle;
the read returned its first bytes in place of the ones asked for.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* sw-client: a read size past u64::MAX reads to the end of the file
offset + size overflowed before the clamp to the file size.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* sw-client: an SSE-encrypted manifest is refused without a decoder
Only filer-encrypted manifests were; an SSE one went to the protobuf
decoder as ciphertext.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* sw-client: an empty write leaves the staged file size alone
A zero-byte write past the end grew the size to its offset, as a
zero-byte pwrite does not.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* sw-client: the chunk cache keeps no empty chunks
They add no bytes, so nothing evicted them and the map grew with every
distinct empty needle, even at capacity zero.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* sw-client: flushing with a zero chunk size is an error, not a panic
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* weed-vector: an empty master list fails at startup
"" split into one empty address that passed the check, and the engine
then waited two minutes for a filer before a generic error.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* lance cluster: an empty master list fails at startup
Cluster::new took "" as one master, so weed-vector, weed-worker and the
CLI waited two minutes for a filer before a generic error.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* direct storage: a failed create keeps its chunks when the filer cannot be asked
The lookup after a failed create is what tells a lost reply from a
refusal; when it fails too, the entry may be saved, so its chunks stay.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* weed-vector: a gateway-described table takes the shared unsigned store options
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* vector.proto: give sse_s3 a new field number and reserve storage_options
Field 4 held the storage_options map on master; a gateway from that build
would send a length-delimited value the engine cannot decode as a bool.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* weed-vector-cli: open s3:// URIs on the cluster's storage
A table named by its s3:// location skipped the store wrapper and opened
Lance's own S3 store, which has no credentials since the fallback flags
were dropped. It now opens like a namespace table and needs --masters;
local paths keep the plain builder.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* vector: an index id is plain path components, and drop trusts describe()
`resolved()` now refuses an id whose part is empty, `.`, `..` or holds a
`/`, since the colocated path would land elsewhere on the filer; and
`drop_index` no longer repeats the colocation check for the location
describe() already enforced for the configured schemes, which made a drop
on a memory:// table fail in the scheme test configuration.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* vector: a file delete takes only its entry; a table path is plain names
The store's delete of one file no longer asks the filer for a recursive
delete, so a path that names a directory does not take its content with
it, and `remove_table` refuses a location whose bucket or path component
is empty, `.`, `..` or holds a separator, which the filer would resolve
outside the table.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* vector: chunks are assigned to the bucket's collection
The filer assigns a chunk's volume from the collection the caller names;
the gateway names the bucket, prefixed by the filer group when the
cluster has one, so a plain assign landed the engine's chunks in the
default collection. The store now asks for `<bucket>` or
`<group>_<bucket>`, with the group taken from the cluster connection.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* vector: a put's content type becomes the entry's mime
Lance sets no content type today, so every entry was written as
application/octet-stream; a put that does carry one now creates the
entry with it, as the gateway does.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* vector: an upload whose create went unanswered is done with
A create the filer took and never answered (a timeout, or a transport
that reports the filer unavailable) may still be applied, so its chunks
are kept, as they already were for a create whose reply was lost. The
upload then records that it is done: a second completion is refused
instead of creating an empty entry, and an abort discards nothing.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* vector: an upload dropped unfinished discards its chunks
A request cancelled mid-write drops its multipart upload with neither
complete nor abort called, leaving its needles on the volume servers.
Dropping an upload that still holds chunks now discards them on the
runtime; a completed upload, or one whose completion is in doubt, holds
none.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* s3: a cluster process registers and deregisters in the vector namespace
The Lance catalog ran a cluster-token caller through the trusted manager
for every operation but the vector-namespace write check of register and
deregister, which still used the caller-checked manager and answered 403
for an index of a LANCE bucket. One `manager(r)` now picks the manager
for all three.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* vector: the worker's flags take two dashes
The worker parses its flags with clap and, unlike the engine, does not
accept Go-style single-dash long flags, so the harness's hardened worker
leg and the README's `-encryptVolumeData` did not start it. The README's
commands also gain the `--masters` the worker now requires.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* vector: the CLI takes --encryptVolumeData
An index build or optimize from the CLI wrote its chunks unencrypted for
the volume servers unless the filer itself asked for the cipher; the CLI
now takes the flag the engine and the worker have.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* vector: a bucket's encryption is read when a table is opened
The setting was cached for a minute, so a bucket recreated with SSE-S3
within that minute had its first tables written plain, and a record that
could not be decoded was read as "no encryption". A table open now reads
the bucket's record from the filer, once per open, and an unreadable
record is an error.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* vector: a removed table's files leave the object cache
`remove_table` asked the cache to forget everything under the table, but
the cache was built without invalidation closures, so the call was refused
and its error dropped: a data file read before the drop was still served
afterwards. The cache now supports the invalidation, and a refusal is an
error.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* vector: an abort keeps the chunks it could not discard
`discard_all` stopped at the first chunk a volume server would not give
up and dropped the rest from the upload's state, so a retried abort found
nothing to do and the needles stayed. Every chunk is now attempted, the
ones left are kept for the next abort or the drop, and the upload can no
longer be completed.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* vector: a chunk write that fails gives its needle back
When the volume server took the chunk but its answer was lost, the needle
stayed under a file id nothing would ever reference. The assignment is
known before the write, so a failed write now discards it, best effort,
before the error is returned.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* vector: the live worker tests require the cluster's masters
With WEED_LANCE_NAMESPACE set and WEED_MASTERS not, the tests seeded their
tables through lance's own S3 store, which reaches nothing, and failed
before a job ran. The missing variable is now named outright.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* direct storage: only the filer's missing-entry error means not found
Matching any "not found" text read failures such as an unresolved remote
storage as a missing entry, so a delete or a listing reported success over
data that was still there.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* direct storage test: compare the chunks' SSE-S3 IVs
Each chunk's metadata wraps its key under a fresh nonce, so the metadata
differed even if the key and IV were reused.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* direct storage: a filer passed over unanswered keeps the outcome open
When one filer timed out on a create and the next refused it, the refusal
came back without the no-answer mark, so the put discarded chunks the
first filer may still reference.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* lance test: start every concurrent writer before awaiting them
The lazy map spawned each writer only after the previous one finished, so
the commits never overlapped.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* direct storage: count failed uploads and deletes
The storage report counted failed reads and lookups but not a volume
upload, an object delete or a table removal that failed.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* S3 Vectors tests: run weed-vector-cli as the cluster's processes run
The CLI ran without the fixture's environment and outside its directory,
so under VECTOR_READ_JWT it could not read the volumes and under
VECTOR_TLS it found no security.toml.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* direct storage: a filer that fell away leaves the outcome open too
An unavailable error (connection reset, deadline, transport failure) can
arrive after the filer took the request, so a refusal from the next filer
no longer settles a create the first one may still apply: the chunks are
kept, as after a timeout.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* direct storage: a chunk write abandoned in flight gives its needle back
The needle is assigned before the write, so a request cancelled while the
write is in flight left it on the volume server with nothing to name it.
`Source::upload` is now `assign` and `write`, and the upload holds the
assignment across the write in a guard: a write that fails discards it
and one abandoned discards it on the runtime, while a write that returns
hands it to the entry.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* lance tests: sign for the namespace as the worker does
The live tests reached the namespace anonymously once the S3 keys went,
and an anonymous caller lists no tables, so detection proposed nothing.
They now take the cluster key from where the worker reads it
(WEED_JWT_FILER_SIGNING_KEY or security.toml) and log at info, so a table
a handler skips says why in the harness output.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* s3tables lifecycle test: run the Lance worker against the cluster
The worker leg of TestLanceTableLifecycle still handed the Rust tests S3
credentials and no masters. The mini cluster now has a cluster key, and
the tests get WEED_MASTERS and that key; passes with
WEED_LANCE_MAINTENANCE=worker (3 fragments became 1, 4 versions removed).
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* direct storage: name no collection outside a filer group, as the gateway
Without a filer group the gateway leaves the collection to the filer, so a
filer.conf rule for the bucket's path picks it; naming the bucket
overrode that rule, and the chunks landed where a bucket delete does not
look.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* direct storage: a read reports the entry's content type
A put now stores its ContentType as the entry's mime, but get_opts
returned no attributes, so the type did not come back on a read.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
* filer.remote.sync: judge the mount table before admitting later events
The unmount was decided inside a processor job, so events behind the
mapping write, such as remote.unmount's purge deletes, were admitted and
could delete remote objects before the exit landed. The stream callback
now reads each mount-table version before the next event, admits nothing
after the one that ends the mount, saves no offset, and drains the
admitted jobs before exiting.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* filer.remote.sync: ignore mount-table versions from before the mount
Without a saved offset the follow starts at the mounted directory's
creation second, so it replays mount-table versions written before
remote.mount added the directory. Each read as an unmount and the
process exited on every start. A version now ends the write-back only
when the version it replaced mapped the directory to its location.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* filer.remote.sync: judge the mount table by the mount's own key
-dir may lie below a mount, and DetectMountInfo resolves it to the
enclosing mount. Looking the table up by -dir found no mapping in any
version, so a replaced table never counted as mapped and the parent's
unmount was missed, admitting its purge deletes.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* filer.remote.sync: a table renamed away ends the mount
A rename of /etc/remote/mount.mapping to another name inside /etc/remote
was ignored, although the filer no longer has a mount table. The table is
now matched by its exact path on both sides of the event, so renaming it
away counts as removing it, and a file of that name elsewhere under
/etc/remote no longer counts as the table.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* filer.remote.sync: a mount on another remote is not the same mount
The mount-table check compared only bucket and path, so remounting the
directory on another remote storage at the same bucket and path kept
writing through the old client, and a replayed unmount from the other
remote stopped the write-back. The remote storage name now counts too.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* filer.remote.sync: check the drain at the exit itself
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* filer.remote.sync: resolve the mount covering -dir in each table version
Judging the table by the startup mount key alone missed a mount added at
-dir itself below that key: the key stayed mapped, so the nested mount and
its later removal were both admitted, and the removal's purge deletes ran
against the startup remote. Each version is now resolved the way
DetectMountInfo resolves the live table, and a different covering mount
ends the follow as a move.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* filer.remote.sync: an earlier mount's unmount still ends the follow
Skipping every version whose replaced table did not map the directory to
this mount also skipped the unmount of an earlier mount of the directory,
which a -timeAgo replay reaches, and admitted its purge deletes against
this mount's remote. Only a version in which no mount covers the
directory, replacing one in which none did, is now skipped as history
from before the mount.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* filer.remote.sync: map paths below -dir under its place in the mount
With -dir below a mount, every remote key was taken relative to -dir and
appended to the mount's location, so /mount/sub/x.txt uploaded to x.txt
instead of sub/x.txt and could overwrite another object. The processor
now gets the remote location of -dir itself; the mount's own location
still judges mount-table changes.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* filer.remote.sync: resume below a mount no earlier than the mount's offset
remote.unmount saves the offset of the directory it unmounts, after its
purge, and leaves the offsets of directories below it alone. A
write-back of -dir=/mount/sub resumed from its own offset after a
remount, replayed the unmount and exited, on every start. Without
-timeAgo it now starts no earlier than the mount's offset. Whatever that
skips was purged, or was synced by a write-back of the whole mount.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* s3tables: refuse a table or view location another record holds
A create, register, update or rename could name a location at, under or
above another catalog record's files, including the files a renamed table
left under its old name. The new record then loaded that table's metadata,
had credentials vended for its files, and could purge them on drop.
Each of those writes now scans the bucket's records and refuses a root
another record claims, and a drop of a record sharing its files is refused.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* iceberg: check a table's location before writing to it
REST create and stage-create write a metadata file, and stage-create
vends write credentials, before the catalog registers the table, so the
create gate now also validates the location and refuses one another
record holds. RegisterTable runs the same gate before it reads the
metadata file, instead of reading first.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* iceberg: keep a dropped location another record holds
DropTable and DropView purge the dropped record's location after the
catalog delete. A view recorded at a table's location took that table's
directory, record included. The purge now leaves a location another
record still holds.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3tables: authorize CreateTable before reporting the namespace
A caller without CreateTable got 404 for a missing namespace, 403 for an
existing one and 500 with decoder text for an unreadable one. The grant is
now evaluated against the bucket first, and only a caller that passes
learns the namespace is missing or unreadable.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* iceberg maintenance: skip a table whose files another record shares
Orphan removal and snapshot expiry delete what a table's metadata does not
reference. For two records over one location that deletes the other's
files, so detection leaves such tables alone.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3tables: count another record's catalog entry as claimed
A table keeping its files elsewhere still owns its entry directory, and a
view recorded there would take that entry with it on drop. The ownership
check now refuses a root at, above or under another record's entry too.
Maintenance still compares files only, so a table created over a renamed
table's old name does not stop that table being maintained.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3tables: check a claimed location again once the record is written
Two creates, or a create and an update moving a root, could both read the
same root free and both land. A create now checks again after writing its
still-marked record and backs out if another record claims the root, and an
update that moved its root checks again after the write and puts the previous
metadata back. Of two racing writers at least one sees the other.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* iceberg: enable stage-create in the location ownership test
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* iceberg maintenance: check for a shared location again before removing orphans
A job can run long after its table was scheduled, or come from a detector
that never checked, so remove_orphans reads the bucket's records itself and
leaves the files alone when another record shares the table's location.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3tables: check a renamed record's new root again once it is written
A record without a location moves its root with its name, and was checked
only before the rename wrote the destination, so a create landing on that
root in between could share it. The rename now checks again with the
destination written and backs out, like a create.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3tables: put a moved record back when its location cannot be checked again
An update that moved its root kept the move when the check after the write
failed to list the bucket, so a racing writer that relied on that check could
share the root. The move is now undone as it is when the root is taken.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3tables: hold a moved record's old root until the move lands
An update moving a record's root wrote the new metadata first and put the
old metadata back when the check that followed found the new root taken.
The old root was free in between, so a create racing for it could pass its
own check and then share it with the restored record.
The update now stores the metadata it is about to write beside the current
one, so the record holds both roots, checks the new root with that
reservation in place, and only then switches the metadata. A move that
backs out drops the reservation without touching the metadata, and an
update that keeps its root clears a reservation a crashed move left behind.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3tables: test a move racing another update of the record
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3tables: drop a move's reservation when a rename or delete takes the record
A rename or a decoupled delete that took a record while an update held its
move reservation left the reservation on the old name, where the move could
no longer clear it, and later creates of that name were refused as taken.
Both now clear it with the record's other attributes, and a failed rename
does not put it back.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* test: register a table from metadata under its own location
The Spark register test registered a live table's metadata file under a
second name, which now fails because both records would own the source's
files. It checks that refusal instead, and registers a copy uploaded under
the new table's own directory, with its location pointing there, to cover
reading and writing a registered table.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* sw-client: the filer and volume client library
The client the Enterprise repository's VFS and vector engine use, with
the same API: filer gRPC for entries, listings, volume locations,
assignments and remote caching; volume HTTP, plain or TLS, for needle
bytes; chunk resolution for sparse, overlapping and manifest chunks; a
location cache; and the write side. Enterprise-only protocol fields
(license, conditional mutations, hard links, KV listing) are left out.
Compiled against weed/pb/filer.proto like the other worker crates.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* weed-lance-worker: reads straight from the filers and volume servers
direct_read::DirectReads wraps the object store Lance opens so that
entries, listings and bytes come from the filers (gRPC) and the volume
servers (HTTP, signed with the read JWT), whatever the entry holds:
inline content, chunk manifests, sparse or overlapping chunks, the
filer's volume encryption, the gateway's SSE-S3 (sse.rs, with the KEK
from security.toml or the filer) and objects tiered to remote storage.
Writes still go to the inner store. The code and its tests are those of
the Enterprise repository's engine.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* weed-vector: read tables from the filers and volume servers
The engine asks the masters for the filers (and follows the list), takes
jwt.signing.read.key, grpc.client, https.client and the SSE-S3 keys from
security.toml, and opens every table with DirectReads innermost and the
read cache outside it. --masters is required: the gateway finds the
engine through them and the engine finds the filers. Writes still go to
the gateway.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* sse: chunks out of a manifest keep their own SSE-S3 keys
A chunk expanded from a manifest carries its id as fid, so keying the
per-chunk key cache by file_id gave every such chunk the first one's key
and IV. Keyed by the chunk's id however it is carried.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* direct reads: a silent filer is passed over like an unreachable one
Each filer attempt is bounded, so a filer that accepts the connection and
never answers does not hold the read or the startup lookup.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* sw-client: carry the assignment's fsync to the volume upload
The filer's storage rule may want a needle on disk before the volume
answers; the assignment says so and the upload now asks for it, as the
Go uploader does.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* weed-vector: argument tests pass the required masters; connect timeout for filers
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* direct reads: a failed read is tried once more against fresh locations
A fetch that fails forgets the volume locations it was made against; the
request now looks them up again and retries instead of failing on a
volume that moved.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* direct reads: the filer deadline bounds a listing page, not the listing
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* weed-vector: security.toml booleans read as viper reads them
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* weed-vector: a filer the master lists badly does not hide the others
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* weed-vector: register with the masters only once ready to serve
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* sw-client: the HTTP transport asks the volume to fsync when the assignment does
write_synced fell back to the trait default, which refuses fsync, so every
upload under a fsync storage rule failed before it was sent.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* sw-client: https.client.ca alone vouches for volume servers
With a CA configured, reqwest still trusted the system roots next to it;
weed processes trust only the configured pool.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* sw-client: an empty volume range is read without a request
A zero length underflowed the Range end, and a range past u64::MAX
overflowed it; the first now returns no bytes, the second an error.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* sw-client: a whole-needle answer to a ranged read is cut to the range
A server or proxy that ignores Range answers 200 with the whole needle;
the read returned its first bytes in place of the ones asked for.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* sw-client: a read size past u64::MAX reads to the end of the file
offset + size overflowed before the clamp to the file size.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* sw-client: an SSE-encrypted manifest is refused without a decoder
Only filer-encrypted manifests were; an SSE one went to the protobuf
decoder as ciphertext.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* sw-client: an empty write leaves the staged file size alone
A zero-byte write past the end grew the size to its offset, as a
zero-byte pwrite does not.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* sw-client: the chunk cache keeps no empty chunks
They add no bytes, so nothing evicted them and the map grew with every
distinct empty needle, even at capacity zero.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* sw-client: flushing with a zero chunk size is an error, not a panic
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* weed-vector: an empty master list fails at startup
"" split into one empty address that passed the check, and the engine
then waited two minutes for a filer before a generic error.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
* lance catalog: in-process DescribeTable for trusted callers
DescribeTable and DescribeTableInternal resolve a table's location and
storage options without the HTTP layer; the internal form runs the lookup
through a trusted manager so the caller's catalog grants do not gate a
resolution its own permission already authorized. Same code as the
Enterprise repository.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* vector.proto: Table carries its location and storage options
The gateway resolves the table it addresses, so the engine can open it
without a describe round trip of its own. Both fields are optional; an
empty location means the engine looks the table up as before.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* lance catalog: test the in-process describe
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* s3vectors: resolve the table before calling the engine
The gateway describes the table in process, through the trusted catalog
manager, and sends its location and storage options with every engine
call. Storage options are vended for the caller's principal, as the
Enterprise router does for queries. A catalog failure answers with the
code the engine would have given.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* weed-vector: open a table at the location the gateway sent
A request whose table carries a location is opened there, with the
storage options the gateway vended over the engine's fallback; the
catalog is described only for a table that comes without one, as from
an older gateway. dataset::open_at and dataset::drop_at take a
description the caller already holds.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* s3vectors e2e: run the engine without a catalog
VECTOR_ENGINE_NO_CATALOG=1 gives the engine an unreachable catalog URL;
every CI leg sets it, so the suite proves the gateway resolves each table
it sends.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* lance catalog: an internal describe may write a vector index
With credential vending on, the options vended to the engine followed the
caller's s3tables grants, which never allow writing the vector namespace,
so every write the gateway sent would have run with read-only credentials
and reads would have lost their version lease. The engine is that
namespace's writer and the caller's vector permission authorized the
operation, so an internal describe of a vector table vends write-capable
options; other namespaces keep the UpdateTable check.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* s3: resolve tables for the engine only while the catalog runs
With -port.lance=0 the gateway has no catalog to resolve through; the
engine then keeps looking tables up itself, as before.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* lance catalog: a deregistered table describes as NoSuchTable
So the gateway answers NotFoundException for it, not InternalFailure.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* weed-vector: trust a sent location only for a colocated vector index
A caller that reaches the engine could otherwise name any storage the
engine's credentials cover. A location is taken from the request only
when the id names a vector index and the location is its colocated path,
which the engine writes already; anything else is described through the
catalog as before.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* weed-vector: scope the read cache by storage endpoint
The cache was keyed by bucket and path alone, so the same path reached
through another endpoint would have shared entries. Each open wraps the
store with the cache scoped to the endpoint its options name.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* weed-vector: keep a configured endpoint over one a request names
A caller that reaches the engine could otherwise send its fallback
credentials to an endpoint of the caller's choosing. When the engine's
--storage-option names an aws_endpoint, a table whose request names
another one is described through the catalog instead.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* weed-vector: use request options only where credentials cannot leave
A request's storage options are taken only when they hold the keys a
gateway's describe produces, so a proxy or an endpoint alias cannot route
the engine's credentials elsewhere, and an endpoint only when it is the
configured one or the engine holds no credentials to send. Anything else
sends the table to the catalog.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* s3vectors e2e: give the engine its endpoint with its credentials
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* weed-vector: a drop checks the request's options like an open
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* weed-vector: an unpinned endpoint needs the request to sign for itself
Without configured credentials the S3 store still signs with the AWS
default chain, so an endpoint from a request is taken only when the
request skips signing or brings its own credentials.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* weed-vector: a trailing slash does not unpin the endpoint, and say so
A configured aws_endpoint written with a trailing slash matched nothing
the gateway sends, so every table fell back to the catalog in silence.
The comparison ignores the slash, and a fallback for a resolved table is
logged at debug.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* weed-vector: resolved by the gateway, or refused
The engine no longer has a catalog: --namespace and --filer-signing-key
are gone, a request whose table names no location is InvalidArgument, and
one whose location is not the index's own or whose storage options the
engine's credentials must not be used with is PermissionDenied. A drop
removes the colocated path it was given.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* s3: the catalog resolves vector tables whether or not its port is served
newLanceServer builds the Lance catalog for the S3 Vectors API; the
listener on -port.lance is optional.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* s3vectors tests: start the engine without a catalog
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* s3: the aux listener test builds the catalog before serving it
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* weed-vector: an unpinned endpoint takes only credentials the request signs with alone
Lance fills an option a request leaves out from the engine's environment,
under any of object_store's names for it (TOKEN, SESSION_TOKEN, ...), and
asks the AWS default chain when the key pair is incomplete. A request that
named its own endpoint with only an access key id, or a key pair while the
engine's environment held a session token, would have sent the engine's
credentials there. Such a request now needs a full key pair, and its own
session token whenever the environment would supply one.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* weed-vector: accept --namespace and --filer-signing-key, ignored
Both flags shipped in 4.52 and are documented, so dropping them made an
existing command line exit at startup with an unexpected-argument error.
They are kept hidden and ignored, with a warning.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3vectors design: the engine is started without a catalog
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* weed-vector: the trust check takes the environment it reads
The options test read the process environment through signs_alone, so a
host exporting AWS_SESSION_TOKEN or TOKEN failed it. The caller now
passes the variable names in; describe passes the engine's own.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* weed-vector: log both endpoints when a request's options are refused
The gateway turns the PermissionDenied into a bare AccessDenied and logs
nothing, so a pinned aws_endpoint written in another form than the one the
gateway sends (host case, a default port, an IP for a name) failed every
call with no trace of which value differed.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3: the Lance catalog sets the credential role whether or not Iceberg runs
-iceberg.credentialRole reached the S3 server only while the Iceberg
listener started, so with -port.iceberg=0 the Lance catalog installed its
vendor but vending stayed off, and the vector engine got no credentials for
the tables the gateway resolves. Both catalogs now set the role through
one helper.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3: set the credential role once, before either catalog serves
Both catalogs wrote the role as they were built; the second write landed
while the first was already serving requests that read it.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
---------
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
* admin: show disk usage on the dashboard instead of the data size
The dashboard's Total Size card summed the volume and EC shard sizes the
volume servers report. That leaves out deleted needles waiting for vacuum,
EC generations kept after a vacuum, and anything else on the disks, so on a
cluster with a lot of EC vacuum activity it showed far less than the disks
held.
The card now shows the bytes in use on the volume servers' disks, the same
statfs numbers the volume server list uses, with the disks' capacity, and
its trend line follows that number. Servers that report no statfs numbers
count their data size, as on the volume server list.
Claude-Session: https://claude.ai/code/session_011UpvafcfUuC2mLxNAXLQJi
* admin: compute the disk usage percentage in float64
current*100 overflows int64 past about 92 PB of used space, so the
dashboard card showed a negative percentage.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* admin: size the volume server disk bar with calculatePercent64
int(host.DiskUsage) truncates byte counts on 32-bit builds, so the bar
width was wrong for any disk over 2 GB there.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* admin: test the cluster disk totals across statfs and fallback servers
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* mount: keep retrying writeback chunk uploads through a volume outage
With -writebackCache, close() returns before the data is uploaded. When
the volume servers stay unreachable past the upload's own retry (~20 s),
the chunk was dropped and the file never appeared, with no error to the
application. Retry transient upload errors with the same bytes until they
succeed, unless the dirty pages were discarded. Permanent errors and the
synchronous mode keep failing as before.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* mount: test writeback upload retries and the cases that still fail
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* operation: keep the filer's read-only verdict on assign failures
UploadWithRetry flattened AssignVolumeResponse errors into a string, so
callers could not tell a read-only rule from an outage. Wrap the filer's
coded error so errors.Is(err, filer_pb.ErrReadOnly) works; the message is
unchanged.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* mount: retry writeback uploads the upload layer reassigns, never read-only
A filer proxy that cannot reach the volume answers 500 with an empty
body, which util.IsTransientError does not match, so that outage still
dropped the chunk. Also retry what operation.ShouldReassignUpload takes.
A read-only rule stays final even when its path or message contains a
transient word such as "unavailable".
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* mount: stop a writeback upload once its dirty pages are discarded
Destroy() was only seen after a failed attempt, so a truncate during the
retry wait still started another upload, and a chunk that finished after
its pages were replaced was attached to the new content. Check before
each attempt, and drop the chunk instead of attaching it; volume.fsck
reclaims it.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* mount: keep Destroy from slipping between the discard check and the append
truncateEntry destroys the pages and then installs the new entry, so a
Destroy landing after the check but before AddChunks put the old chunk
into the replacing entry. Hold one lock across the check and the append,
and take it in Destroy while setting the flag.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* mount: leave the Windows interrupt drain to the capped filesystem hook
Interrupt hooks run in reverse registration order, so the Windows hook ran
before the filesystem hook and waited on pending writeback flushes without a
bound. With writeback uploads now retrying through an outage, Ctrl-C could
wait until the volume servers came back. Unmount only, as on Linux, and let
the filesystem hook drain for at most 30 s and keep the write cache.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* mount: test a writeback upload whose assign keeps timing out
With every volume server gone the assign RPC times out until the filer
client gives up; that error must be retried like a refused upload.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* grafana: chart the remote-deletion tombstone gauges
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* grafana: describe the fetch hold that outlasts a full tombstone table
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* grafana: word the tombstone panel description like the gauge help
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* deps: bump seaweedfs/raft so the leader quorum check follows the election timeout
A single master resuming raft state self-elects on a 1 ms election
timeout and restores the configured one afterwards, but the leader kept
running its quorum check every 1 ms for the rest of the process. The
raft change re-arms that check from the current timeout.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* deps: bump seaweedfs/raft to the raft PR head with the extended test
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* deps: bump seaweedfs/raft for the non-positive election timeout guard
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* deps: bump seaweedfs/raft so a timeout change re-arms the leader quorum check
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* deps: pin seaweedfs/raft v1.2.4
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* filer: fold small chunks into manifests 256 at a time
Every metadata event carries the file's whole chunk list, so a file grown
by small appends re-sent each chunk record on every append until the
list reached ManifestBatch (10000). Chunks under 1 MiB now fold once 256
of them build up; full-size chunks still fold at ManifestBatch, so large
files keep their flat lists and read without a manifest fetch.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* filer: merge appended small chunks at the small-chunk fold count
The append path merges small chunks into full-size ones, but only at
1000 of them; the fold now hides small chunks in a manifest at 256, so
the merge would never see enough of them. Merging at the same count
keeps rewriting them into full chunks first.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* filer.remote.gateway: keep the remote object while a version under a delete marker is locked
Every version of a versioned bucket uploads to the object's key, so on a
remote without versioning that object is the only remote copy of the
latest version. A delete marker deleted it even when a version still held
a legal hold or unexpired retention. Keep it while any version of the key
is locked, the same rule the filer uses before pruning a remote entry.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* filer.remote.gateway, filer.remote.sync: say that earlier versions need remote versioning
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* filer.remote.gateway: delete a kept delete marker's object once the version lock is released
A delete marker kept for a locked version stays unstamped. When an
update or delete leaves a version that carried a legal hold or retention
without an active lock, look up the key's latest version and, if it is
still that unstamped marker, run the delete again. Only changes to
versions that carried a lock trigger it, so the noncurrent mark and tag
updates of unlocked versions do not send repeat remote deletes.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* filer.remote.gateway: test the kept delete marker retry through the bucket mount
The retry reads the key's versions through a filer client parameter, as
the rename helpers do, so a test can check the mapped object key and the
marker stamp, and that lock-releasing updates and deletes reach it.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* filer.remote.gateway: check the lock release lookup goes to the gateway's filer
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* filer.meta.backup: stream from the start of an unfinished walk
A walk only inserts entries, so when the first walk is interrupted the
next start walks again and streams from the second walk's start: deletes
and renames between the two walks never reach the backup store. Keep the
first walk's start time in the store until a walk completes and stream
from it.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* filer.meta.backup: start fresh once the stream has passed an earlier walk's start
A walk start left in the store by a crash between saving the offset and
clearing it, or by an interrupted -restart followed by a plain restart,
stops mattering once the stream saves an offset past it. Ignore it then
instead of replaying from it on the next walk.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* filer.meta.backup: fail the walk on an unreadable offset
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* filer: a full remote-deletion tombstone table keeps the lazy fetch off
At the cap a new delete under a remote mount was dropped with a log line
and isRemoteDeletionPending answered false for it, so the lazy remote fetch
resurrected deleted paths. While the table is full every path may be a
pending delete: isRemoteDeletionPending answers true until the table has
room again. The size and the full state are gauges
(filer_remote_deletion_tombstones, filer_remote_deletion_tombstones_full).
(cherry picked from commit f3be0ab965cde3bfd3053d3ac720fbbe3f68e98b)
* filer: a full tombstone table drains on reads and lifts after the unrecorded deletes age out
While the table is full, a lookup miss evicts expired tombstones and those
the write-back offset has passed, so the table drains without a mutation.
The full state lifts only once the table has room and the newest
unrecorded delete is past the TTL: an unrecorded delete has no event to
confirm it, so only the TTL makes it safe, as for a pending tombstone.
(cherry picked from commit 19394fe5d67798af7b0d755ac18c83c75e20a12b)
* filer: a full tombstone table drains per mount, only when its offset advances
Offset-based release applies to the tombstones under the mount whose
offset was read; another mount's tombstones answer to their own offset. A
read sweeps the table only once that offset advanced since the last drain
for the mount or the unrecorded deletes aged past the TTL, so a listing of
many missing children does not rescan the table per child.
(cherry picked from commit 9cf00e2e54d70e6ba530e6ec1b1ff4b502873847)
* filer: the tombstones-full gauge help names when it clears
---------
Co-authored-by: james <james@medable.com>
* filer: no lazy remote fetch inside a store transaction
A rename onto an existing target deletes the target inside the store
transaction, then CreateEntry's FindEntry misses the row and runs the lazy
remote fetch, which persists the archive copy on a second connection. The
store wrapper drops that call's deadline, so the UPDATE waits without bound
on the row lock of the transaction's own uncommitted DELETE while the
rename goroutine holds the path's exclusive locks: every later lookup,
rename and delete of the path on that filer waits forever.
A lookup whose context carries the store transaction skips the fetch.
(cherry picked from commit b69e9535c2597d0c8e856f031d3778fddfd4159c)
* filer: materialize remote-only rename paths before the transaction
With the lazy fetch off inside a store transaction, a source or target the
filer holds only on the remote would read as absent to AtomicRenameEntry
and StreamRenameEntry: a rename with FailIfTargetExists could overwrite it,
and a remote-only source would skip the deferred copy. PrefetchRemoteEntries
runs the fetch for both paths before BeginTransaction, under the path locks.
(cherry picked from commit e61f9718018669ef59768cd5eda25b6466d69597)
* filer: skip the lazy remote fetch only in CreateEntry's lookup inside a transaction
The guard on every lookup inside a store transaction hid a source or target
the filer holds only on the remote from the rename, and the prefetch before
the transaction did not reach the children of a directory rename. The
deadlock needs a write: CreateEntry's lookup after the transaction deleted
the path. That lookup alone skips the fetch; reads inside the transaction
keep it, and their persist on a second connection holds no lock the
transaction waits for.
(cherry picked from commit f555092c324780a8fe5314b238694551e3ef6be5)
* filer: mark a store transaction independently of the store's own key
ArangoDB and FoundationDB keep their transaction under other context
keys, so the CreateEntry guard keyed on "tx" missed them. The filer now
marks every context returned by BeginTransaction and the guard checks
that marker.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
(cherry picked from commit bd0505eb80)
* filer: keep the remote check for an exclusive create in a transaction
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
(cherry picked from commit b6aac15c3b)
* filer: test that a create outside a transaction still consults the remote
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
(cherry picked from commit 972daddd84)
---------
Co-authored-by: james <james@medable.com>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* s3api: mint a versioned PUT's id after the object write lock
putVersionedObject minted the version id before the upload and the lock
wait. A DELETE that committed while the PUT waited got a newer id, so a
conditional PUT whose If-None-Match passed against the delete marker was
stored behind it: accepted, but GET returned NoSuchKey.
The finalize now re-mints the id once the lock is held and the
precondition passed, so the version orders after everything the check
saw. An ambiguous routed attempt keeps its id, since recovery looks for
that entry.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3api: mint a routed versioned PUT's id after the upload
On the routed path the version id was still the one minted before the
upload, and RECOMPUTE_LATEST picks the newest name. A DELETE that
committed during a long upload left its marker latest over the accepted
PUT. Re-mint right before the routed transaction, as the lock path does.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* filer: stop unbounded metadata-log growth on append-only files
Two independent leaks make a filer metadata log grow far faster than the
data it describes. On a cluster whose live logs weigh a few hundred
megabytes, they produced tens of gigabytes of metadata per day.
1. Metadata events embed the whole old and new entry, chunk lists
included. An append-only file — a log or a journal written in small
pieces — therefore re-serializes every chunk it ever had on every
append, so the cost is quadratic in the number of chunks. A
2,700-chunk file emits a couple of megabytes per append. Cap the
carried chunk list at 256 entries and set
metadata_event_chunks_truncated so subscribers that need the content
read the entry from the filer instead of trusting the event.
Subscribers that only track names, attributes, or chunk deltas keep
working unchanged.
2. appendToFile uploads the chunk before committing the filer entry.
When the commit fails — store error, cancelled context, filer
shutting down — the chunk is stranded: no entry references it, so no
later delete can mark it, and neither vacuum nor fsck can see it
because both work from the entry index. The volume server keeps
counting those bytes as live data forever. Release the chunk when
the commit fails.
* filer: keep chunk lists in metadata events
Mount meta caches, peer filers replaying into their own store, and the
filer.sync, filer.backup and filer.meta.backup sinks all store the event's
entry as the file. Dropping the chunk list past a count left those copies
of a large file without content.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* filer: release a metadata log chunk the lookup or commit leaves unreferenced
The flush retries a failed append with a new upload, so a store outage
left one stranded chunk per retry. A failed lookup strands the chunk too.
Use DeleteUncommittedChunks, as the other write paths do: the chunk was
never part of the file, so path chunk-deletion rules do not apply.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* filer: keep a metadata log chunk the failed create stored
CreateEntry inserts a new entry before its parent folders, so a parent
failure returns an error with the entry already referencing the chunk.
Release it only when the store shows no entry, or an entry without it.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* filer: count a metadata log piece the failed create stored as appended
The flush retry would append the same bytes after it, and a piece cut
inside a record then breaks replay of that segment. The retry takes the
update path, which does not create parent folders either, so returning
the error only duplicated the piece.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Martin Morez <martin@morez.org>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* s3 lifecycle: send the listed entry's identity on walker deletes
The daily walker dispatched LifecycleDelete with no ExpectedIdentity, so
the server skipped its compare-and-swap and an object rewritten between
listing and dispatch was deleted. The list adapter now records each
listed entry's identity, computed the same way as on the event path, and
the walker sends it.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3 lifecycle: current-version expiration must target the latest version
The walker addresses the latest version of a versioned key by id. A
version file never changes, so its identity still matches after a newer
PUT, and the delete went ahead: a delete marker over the new version with
versioning enabled, or the new null version deleted while suspended.
LifecycleDelete now resolves the key's latest version for
EXPIRATION_DAYS/EXPIRATION_DATE requests that carry a version id and
answers NOOP_RESOLVED (VERSION_NOT_LATEST) when it is a different one.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3 lifecycle: check the latest version for unaddressed suspended expirations
With versioning suspended, an expiration without a version id reads the
plain entry, which is the null version but not necessarily the latest:
a newer real version can sit in .versions. The latest-version check now
runs in the expiration branch, by versioning state: for every suspended
request (an unaddressed one addresses the null version) and for enabled
requests that carry a version id.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3 lifecycle: test that each version sibling carries its own identity
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* storage: skip quarantined EC shards on local reads
A needle read tried the local shard first even when its disk was
quarantined after repeated IO errors, so every read paid the failing
disk's EIO latency before falling back. Treat such a shard as not local
so the read goes to peers or reconstruction, and read it only as a last
resort when no healthy copy answers. Scrub still reads it.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* volume: refuse EC shard reads from a quarantined disk
VolumeEcShardRead served peers from a shard whose disk was quarantined,
and it ignored the .ecx lookup error: a failed index read returns a
tombstone size, so the peer was told a live needle was deleted. Refuse
quarantined shards and fail the request on an index read error; a
needle missing from the index still reads, as before.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* storage: report quarantined EC shards deleted right away
Quarantining an EC volume only dropped it from later full EC heartbeats,
so the master kept routing reads to the faulty disk for up to 17
heartbeat intervals. The heartbeat that quarantines the volume now
carries its shards as deleted, once.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* storage: do not announce EC shards mounted onto a quarantined volume
MountEcShards and the shard load notifier reused a quarantined EcVolume
and announced the new shard, pointing the master back at the faulty disk
until the next full EC heartbeat. Skip the announcement, and fail the
mount so a caller moving a shard here keeps its source copy.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* seaweed-volume: count IO errors on local EC shard reads
Go's readLocalEcShardInterval feeds every local shard read into the EC
volume's IO error streak; the Rust needle read and recovery gather did
not, so a failing disk under local reads never moved toward quarantine.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* seaweed-volume: skip quarantined EC shards on local reads
Mirror the Go read path: a needle read and the recovery gather treat a
shard on a quarantined disk as not local, and read it only as a last
resort once peers and reconstruction fail. Scrub still reads it.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* seaweed-volume: refuse EC shard reads from a quarantined disk
Mirror Go's VolumeEcShardRead: answer Unavailable for a shard whose disk
is quarantined, so the peer tries the next holder or reconstructs.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* seaweed-volume: report quarantined EC shards deleted right away
Mirror Go: the volume heartbeat that quarantines an EC volume carries its
shards as deleted, once, instead of leaving the master to drop them at
the next full EC heartbeat.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* seaweed-volume: fail EC shard mounts onto a quarantined volume
Mirror Go's MountEcShards: the heartbeat already leaves such a shard
unannounced, so fail the mount and let a caller moving the shard here
keep its source copy.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* storage: keep reading EC needles when the shard lookup fails
A failed or short master lookup aborted the read before any shard was
tried, so the quarantined-disk last resort never ran once the master had
dropped those shards, and local shards went unread whenever the master
was unreachable. Read on with the cached locations, as the Rust server
already does.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* ec: log a failed last-resort read of a quarantined shard
When reconstruction and the quarantined read both fail, say why the local
read failed too. Go and Rust.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* ec: drop a shard whose mount was refused on a quarantined disk
The refused mount left the new shard loaded on the quarantined volume,
where it was neither served nor announced, and the in-memory claim drew
later copies of that shard back to the same disk. Unload it again,
keeping a shard that was already mounted there. Go and Rust.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* ec: finish a mount batch before reporting a failed shard
A shard that failed to mount, now including one refused on a quarantined
disk, returned before the sidecar reload, and in Rust before the
heartbeat notify, so shards mounted earlier in the same batch kept
checksum protection off and reached the master only at the next full EC
heartbeat. Stop at the failed shard but finish the rest first. Go and
Rust.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* ec: keep a peer's deleted answer when its stream fails after it
A shard read recorded the holder's deleted answer, and a stream error
after it failed the read: Rust discarded the flag, and Go let the next
source or the reconstruction result overwrite it. With the last-resort
read of a quarantined shard, that could serve a needle a peer had
reported deleted. Once the deleted answer is in, end the read on a later
stream error instead of failing it.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* seaweed-volume: use the platform media error in the quarantine heartbeat test
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* storage: rebuild EC intervals from quarantined shards when healthy ones fall short
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* seaweed-volume: rebuild EC intervals from quarantined shards when healthy ones fall short
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* volume: serve a peer's EC shard read from a healthy copy on another disk
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* seaweed-volume: serve a peer's EC shard read from a healthy copy on another disk
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* seaweed-volume: refuse a quarantined EC shard read instead of serving a sibling copy
A sibling disk's copy can have its own deletion journal, and an EC blob
delete is journaled on one local copy only, so the sibling could serve a
deleted needle.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* volume: refuse a quarantined EC shard read instead of serving a sibling copy
A sibling disk's copy can have its own deletion journal, and
VolumeEcBlobDelete journals a delete on one local copy only, so the
sibling could serve a deleted needle.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* seaweed-volume: check the disk an EC shard mounted on for quarantine
mount_ec_shard now returns the disk it mounted on, and the mount RPC
checks only that disk, like Go's MountEcShards. Before, a quarantined
sibling holding the same shard failed the mount while the healthy
disk's copy stayed mounted.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* storage: test an EC mount beside a quarantined sibling
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* seaweed-volume: check the quarantined-sibling mount test by its result
A server that remounts a shard on the disk already claiming it refuses
the mount there instead. The test now only requires that the result
matches whether the healthy disk ended up holding the shard.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* storage: check the quarantined-sibling mount test by its result
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* storage: expect the quarantined-sibling mount result from the disk the store picks
The test accepted a refused mount whenever the healthy disk stayed empty,
so a handler that refused before mounting passed too. Ask the store which
disk it picks first, then require success exactly when that is the healthy
disk.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* seaweed-volume: expect the quarantined-sibling mount result from the disk the store picks
Mirrors the Go test. add_location's cross-disk reconcile mounted shard 1 on
the healthy disk too, so the test now unmounts it there first, leaving the
quarantined sibling as the only claimant, as in Go.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* filer.remote.sync: bounded filer lookups, one chunk cache per process
The filer lookups on the sync path (currentEntry, isSuperseded,
uploadCurrentEntry, liveRemoteEntry, the mount directory read) ran with no
deadline. A lookup that never answers keeps its job alive, the job keeps
its path in the MetadataProcessor conflict index, AddSyncJob waits on the
condition and the daemon stops consuming events with no log line. Each
lookup now carries a 30 s deadline, so the job fails and the processor's
stop-and-resubscribe path runs.
remoteUploadReader built a chunk cache per upload; the cache's worker
goroutine is never stopped, so each upload leaked one goroutine and its
cache (21k goroutines, ~150 MB/h under lock-file churn, OOM at the memory
limit). One chunk cache per process.
(cherry picked from commit ff77fd222772d4f3b68f34d5ee39a66684f90e78)
* filer.remote.sync: a failed mount lookup at startup never resolves to now
collectLastSyncOffset returned time.Now() when the mounted directory lookup
failed, which skips every event since the saved offset. The saved offset is
used instead; without one the daemon exits so a restart retries.
(cherry picked from commit 8fd3124f5b3d3f8c3a291263a710047b4aff81ae)
* filer.remote.sync: no upload on an unanswered supersession lookup; bounded offset read
retriedWriteFile returns the supersession lookup's error instead of
treating an unanswered lookup as "not superseded": in a replay an older
snapshot could otherwise overwrite a newer remote object. The processor
retries the event. isSuperseded keeps its boolean form for the other call
sites. GetSyncOffset's KvGet carries a 30 s deadline so a startup that falls
back to the saved offset cannot hang on it.
TestGatewayUncachedRewriteIsStillUploaded asserted the upload on a failed
lookup; it now asserts the withheld upload and the lookup error.
(cherry picked from commit 8fc2e718f7cd8322615b4e40007be903ade2dbfa)
* filer.remote.sync test: keep the slices import the other gateway tests use
* filer.remote.sync: an unanswered lookup after a failed attempt ends the retry loop; an unread offset never resolves to now minus timeAgo
After a failed upload attempt the supersession lookup's error ended in
isSuperseded's boolean and the loop could upload the snapshot again
unverified. The lookup error now ends the loop (errLookupUnanswered, not
retried) and the processor retries the event. With -timeAgo set, a failed
read of the saved offset resolved to now minus timeAgo, skipping everything
older; the daemon exits instead so a restart retries.
(cherry picked from commit 55ca2a88e01d7b556d494a404b9cc263c27b2566)
* filer.remote.sync: an unresolved start offset returns to the retry loop instead of exiting
resolveLastSyncOffset returns the lookup or offset-read error; both callers
(directory and buckets) hand it to util.RetryUntil, which retries with
backoff instead of the process exiting.
* filer.remote.sync: keep doc comments on the functions they describe
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* filer.remote.sync: isSuperseded returns the lookup error itself
Its only production callers needed the error, so the boolean wrapper was
left to the tests. The tests now assert the error on a failed lookup.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* filer.remote.sync: bound the stamp written after each upload
updateLocalEntry's UpdateEntry was the last filer call on the sync path
without a deadline. A filer that never answers the stamp keeps the job and
its path in the conflict index the same way an unanswered lookup did. The
stamp carries IF_ENTRY_EQUAL, so a retried event after a timeout either
stamps again or skips a stamp that already landed.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* filer.remote.sync: an unread offset never resolves to the mount's creation time
With the offset read now bounded, a timed-out read fell through to the
directory's creation time and replayed every event already synced. The
error goes back to the startup retry loop, as with -timeAgo; a missing
offset still reads as zero without an error.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* filer.remote.sync: stamp a copy of the event entry
updateLocalEntry set the RemoteEntry and dropped the rename marker on the
caller's entry before the RPC. A transient stamp failure retries the same
event in place, which then read as already synced and skipped both the
upload and the stamp. The stub filer now applies a successful update the
way the filer does, since tests had relied on that shared object.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* filer.remote.sync: a directory not created yet starts now, as before
A gateway started before the first bucket looked up a missing buckets
directory, had no saved offset, and returned an error to the retry loop
on every attempt until a bucket appeared. A missing directory with no
saved offset has nothing to replay, so it starts now as it did before.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* filer.remote.sync: take the start time before the mount lookup
A bucket created between a not-found lookup and the end of the offset
read had its creation before a start taken afterwards, so the gateway
never created its remote bucket.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* filer.remote.sync: save the resume position when a subscription ends
A failed event ends the subscription with ErrResubscribe, and the retry
loop runs startup again, recomputing -timeAgo from a later now. The
periodic checkpoint only runs as events arrive, so a failure before the
first one could start the retry after the event it failed on. The resume
position is now saved whenever the subscription returns.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* remote_storage: bound the sync offset write like the read
The offset write runs in the subscription's checkpoint callback and, now,
when a subscription ends; a filer that never answers it stalled either.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* filer.remote.sync: say that any saved offset wins over the creation time
The creation time in seconds was compared with the offset in
milliseconds, which made any saved offset win. That is the behaviour to
keep: remote.unmount recreates the directory and then saves the offset,
usually in the same second, so a seconds comparison would start at the
creation second and replay the purge. The comparison now states it.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* filer.remote.sync: keep the resume position for the retry when its write fails
The retry in the same process now resumes from the position the ended
subscription returned, as filer.backup does, so a failed offset write
followed by an empty read cannot recompute -timeAgo past a failed event.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* filer.remote.sync test: give the same-second saved offset a fractional part
With whole seconds, returning the creation second resolved to the same
time and passed.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: james <james@medable.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* filer.remote.sync: bounded filer lookups, one chunk cache per process
The filer lookups on the sync path (currentEntry, isSuperseded,
uploadCurrentEntry, liveRemoteEntry, the mount directory read) ran with no
deadline. A lookup that never answers keeps its job alive, the job keeps
its path in the MetadataProcessor conflict index, AddSyncJob waits on the
condition and the daemon stops consuming events with no log line. Each
lookup now carries a 30 s deadline, so the job fails and the processor's
stop-and-resubscribe path runs.
remoteUploadReader built a chunk cache per upload; the cache's worker
goroutine is never stopped, so each upload leaked one goroutine and its
cache (21k goroutines, ~150 MB/h under lock-file churn, OOM at the memory
limit). One chunk cache per process.
(cherry picked from commit ff77fd222772d4f3b68f34d5ee39a66684f90e78)
* filer.remote.sync: a failed mount lookup at startup never resolves to now
collectLastSyncOffset returned time.Now() when the mounted directory lookup
failed, which skips every event since the saved offset. The saved offset is
used instead; without one the daemon exits so a restart retries.
(cherry picked from commit 8fd3124f5b3d3f8c3a291263a710047b4aff81ae)
* filer.remote.sync: no upload on an unanswered supersession lookup; bounded offset read
retriedWriteFile returns the supersession lookup's error instead of
treating an unanswered lookup as "not superseded": in a replay an older
snapshot could otherwise overwrite a newer remote object. The processor
retries the event. isSuperseded keeps its boolean form for the other call
sites. GetSyncOffset's KvGet carries a 30 s deadline so a startup that falls
back to the saved offset cannot hang on it.
TestGatewayUncachedRewriteIsStillUploaded asserted the upload on a failed
lookup; it now asserts the withheld upload and the lookup error.
(cherry picked from commit 8fc2e718f7cd8322615b4e40007be903ade2dbfa)
* filer.remote.sync test: keep the slices import the other gateway tests use
* filer.remote.sync: an unanswered lookup after a failed attempt ends the retry loop; an unread offset never resolves to now minus timeAgo
After a failed upload attempt the supersession lookup's error ended in
isSuperseded's boolean and the loop could upload the snapshot again
unverified. The lookup error now ends the loop (errLookupUnanswered, not
retried) and the processor retries the event. With -timeAgo set, a failed
read of the saved offset resolved to now minus timeAgo, skipping everything
older; the daemon exits instead so a restart retries.
(cherry picked from commit 55ca2a88e01d7b556d494a404b9cc263c27b2566)
* filer.remote.sync: an unresolved start offset returns to the retry loop instead of exiting
resolveLastSyncOffset returns the lookup or offset-read error; both callers
(directory and buckets) hand it to util.RetryUntil, which retries with
backoff instead of the process exiting.
* filer.remote.sync: keep doc comments on the functions they describe
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
(cherry picked from commit 952d80c4d3)
* filer.remote.sync: isSuperseded returns the lookup error itself
Its only production callers needed the error, so the boolean wrapper was
left to the tests. The tests now assert the error on a failed lookup.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
(cherry picked from commit 51f603b5ec)
* filer.remote.sync: bound the stamp written after each upload
updateLocalEntry's UpdateEntry was the last filer call on the sync path
without a deadline. A filer that never answers the stamp keeps the job and
its path in the conflict index the same way an unanswered lookup did. The
stamp carries IF_ENTRY_EQUAL, so a retried event after a timeout either
stamps again or skips a stamp that already landed.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
(cherry picked from commit c7338fb68f)
* filer.remote.sync: an unread offset never resolves to the mount's creation time
With the offset read now bounded, a timed-out read fell through to the
directory's creation time and replayed every event already synced. The
error goes back to the startup retry loop, as with -timeAgo; a missing
offset still reads as zero without an error.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
(cherry picked from commit 0a1b6c2944)
* filer.remote.sync: stamp a copy of the event entry
updateLocalEntry set the RemoteEntry and dropped the rename marker on the
caller's entry before the RPC. A transient stamp failure retries the same
event in place, which then read as already synced and skipped both the
upload and the stamp. The stub filer now applies a successful update the
way the filer does, since tests had relied on that shared object.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
(cherry picked from commit 1f332e5866)
* filer.remote.sync: a directory not created yet starts now, as before
A gateway started before the first bucket looked up a missing buckets
directory, had no saved offset, and returned an error to the retry loop
on every attempt until a bucket appeared. A missing directory with no
saved offset has nothing to replay, so it starts now as it did before.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
(cherry picked from commit c31497a9a3)
* filer.remote.sync: take the start time before the mount lookup
A bucket created between a not-found lookup and the end of the offset
read had its creation before a start taken afterwards, so the gateway
never created its remote bucket.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
(cherry picked from commit f535fca8f6)
* filer.remote.sync: save the resume position when a subscription ends
A failed event ends the subscription with ErrResubscribe, and the retry
loop runs startup again, recomputing -timeAgo from a later now. The
periodic checkpoint only runs as events arrive, so a failure before the
first one could start the retry after the event it failed on. The resume
position is now saved whenever the subscription returns.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
(cherry picked from commit 7be919ded1)
* remote_storage: bound the sync offset write like the read
The offset write runs in the subscription's checkpoint callback and, now,
when a subscription ends; a filer that never answers it stalled either.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
(cherry picked from commit f888ac6b8f)
* filer.remote.sync: say that any saved offset wins over the creation time
The creation time in seconds was compared with the offset in
milliseconds, which made any saved offset win. That is the behaviour to
keep: remote.unmount recreates the directory and then saves the offset,
usually in the same second, so a seconds comparison would start at the
creation second and replay the purge. The comparison now states it.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
(cherry picked from commit a0d60ee00f)
* filer.remote.sync: keep the resume position for the retry when its write fails
The retry in the same process now resumes from the position the ended
subscription returned, as filer.backup does, so a failed offset write
followed by an empty read cannot recompute -timeAgo past a failed event.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
(cherry picked from commit 39ef551bcf)
* filer.remote.sync test: give the same-second saved offset a fractional part
With whole seconds, returning the creation second resolved to the same
time and passed.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
(cherry picked from commit 4bb74382a6)
---------
Co-authored-by: james <james@medable.com>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* volume: a delete cancels the writes of its needle still in flight
A write queued behind the volume locks, for example behind another write's
fsync on a busy disk, is not in the needle map yet. A delete of that needle
in the meantime finds nothing, and the write lands afterwards, so the needle
comes back on this copy only while the other replicas have it deleted.
Each volume now tracks its writes from entry until they return. A delete
cancels the ones for its needle, and a cancelled write that landed is
tombstoned before it returns, unless the needle was deleted or rewritten
since.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* volume: a replicate delete that finds nothing cancels the pending write
Rolling back a failed chunk upload sends DELETE ?type=replicate to every
holder, including the slow one whose write timed out on the client. That
holder has not indexed the write yet, so the delete answered 404 and the
write landed live afterwards. Cancel the needle's pending writes, then look
again in case one finished in between. A plain delete that finds nothing
still answers 404 without cancelling: it is not sent on to the replicas.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* seaweed-volume: a delete cancels the writes of its needle still in flight
Mirrors the Go volume. Volume::track_pending_write registers a write until
its guard drops, delete_needle cancels the needle's pending writes, and
settle_pending_write tombstones a cancelled write that landed unless the
needle was deleted or rewritten since.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* seaweed-volume: track uploads and cancel them on a replicate delete
An upload is tracked from the handler, and a durable one from submission
to the write queue, where the guard travels with the request: a handler
dropped while the write waits in the queue does not untrack it, and the
worker settles it after the batch. A replicate delete that finds nothing
cancels the needle's pending writes and looks again, like the Go handler.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* volume: take a cancelled write back in the lock span that publishes it
The write tombstoned itself only after it returned, with the volume locks
released. In that gap a compaction could move the record so the offset
check skipped it, a close could drop it, and an identical write sent after
the delete could match the live record and report success before the
tombstone removed it. The check now runs right after the publish, under
the same dataFileSyncLock and dataFileAccessLock hold, in the plain, fsync
and batch write paths.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* volume: a delete cancels only the writes carrying its cookie
A delete with the right needle id but the wrong cookie could cancel an
in-flight write and get it tombstoned. Each pending write now records its
cookie, and a cancel only marks the matching ones.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* seaweed-volume: take a cancelled write back in the lock span that publishes it
Mirror the Go change: settle_pending_write ran after write_needles_grouped
returned, so a compaction, a close or an identical write later in the same
batch could slip in before the tombstone. drop_cancelled_write now runs
right after the publish, under the same sync and access lock hold, in the
synced run, the unsynced run and write_needle_unsynced_or_refused.
write_pending_needles carries each write's PendingWrite down to that
point.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* seaweed-volume: a delete cancels only the writes carrying its cookie
Mirror the Go change: a delete with the right needle id and the wrong
cookie no longer cancels an in-flight write. Each tracked write keeps its
cookie next to its cancel flag.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* seaweed-volume: refuse a queued write whose volume was remounted
The write queue tracks a write on the volume found at submit, but the
worker looks the volume up again by id. After a remount, a delete reaches
the new volume and cannot cancel the write registered on the old one, so
the worker now refuses such a write instead of landing it on the new
volume.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* seaweed-volume: decode the delete's type parameter like Go's FormValue
The delete handler matched the raw query text for type=replicate, so a
percent-encoded name or value was not treated as a replica delete and did
not cancel the write it undoes. Parse the query the way the upload handler
does.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* volume: test a replica delete landing in a write's fsync window
Drive DeleteHandler while a durable write is parked in its .dat fsync,
appended but not indexed: the replica delete answers 404 and the write is
taken back when it publishes, while a plain delete leaves it live.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* volume: flush a take-back tombstone before the index row that points at it
A durable write taken back during its fsync appends the tombstone after the
.dat sync, and the .idx sync that follows made its row durable while the
record itself could still be lost, which reloads the volume read only. The
fsync and batch paths now sync the .dat again first when a take-back
appended a tombstone; a failure stops the volume taking writes like a failed
index sync.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* seaweed-volume: flush a take-back tombstone before the index row that points at it
A durable write taken back in write_synced_run appends its tombstone after
the .dat sync, so the .idx flush that follows could make the row durable
while the tombstone record itself was still in the page cache. The run now
syncs the .dat again first when a take-back appended a tombstone; a failure
stops the volume taking writes like a failed index flush.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* volume: a skip-cookie batch delete cancels the writes of the file id it names
BatchDelete with SkipCookieCheck left the needle's cookie at zero, so the
delete's cancel matched none of the in-flight writes and a retry queued
behind it could land afterwards. The non-EC delete now carries the file
id's cookie, as the Rust volume server already does; the EC path keeps
cookie 0, which DeleteEcShardNeedle reads as "skip the check".
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* volume: release a held test write before the store closes
A failed wait ended the test before release ran, so the writer stayed parked
in its fsync and the store cleanup blocked on its lock until the package
timed out.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* s3tables: route REST RenameTable and refuse unrouted S3 Tables requests
PUT /tables/<arn>/<ns>/<name>/rename had no route, so it fell through to the
/{bucket} routers and ran as a PutObject into a bucket named "tables",
answering 2xx while the table kept its name.
- Answer any s3tables-signed request no S3 Tables route takes with 404
UnknownOperationException, before the health, domain and bucket routers.
- Route the AWS RenameTable REST call (newNamespaceName, newName, both
optional; versionToken).
- Honor versionToken on RenameTable, checked against the snapshot the claim
asserts so a commit between the two reads cannot slip a stale token through.
* s3tables: match the credential scope's service, not a region named s3tables
The unrouted-request fallback widens what a false match costs: an S3
client whose region is named s3tables had every request answered as an
unknown S3 Tables operation.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3tables: refuse an empty newName or newNamespaceName on REST rename
An empty field was read as omitted, so {"newNamespaceName":"other",
"newName":""} moved the table under its old name instead of failing
validation the way AWS does.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3tables: check the rename version token after the snapshot comparison
Once the separately read metadata is known to equal the claimed snapshot,
the token compares against the parsed metadata; no second decode of the
snapshot bytes is needed.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3tables test: cover policy and tag changes in the rename snapshot compare
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3tables test: give each REST rename subtest its own source table
The token subtests renamed the table the first subtest produced, so run
alone they failed on a missing table instead of testing the token.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* master: keep the hostname out of raft metric names
go-metrics prefixes gauge names with the hostname by default, which gives
every master its own metric family, e.g. SeaweedFS_<host>_raft_peers.
Prometheus already tells masters apart by the scrape target.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* master: send Hashicorp raft metrics through the go-metrics global raft uses
hashicorp/raft and raft-boltdb report through github.com/hashicorp/go-metrics,
but the master set up its prometheus sink on the armon/go-metrics global,
so a -raftHashicorp master exported no raft metric families. The sink is now
registered on the hashicorp/go-metrics global, and armon/go-metrics is no
longer a dependency.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* grafana: chart raft commit time and leader contact
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* grafana: keep the cluster in the raft panels
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* grafana: average raft commit time and leader contact over the rate window
The go-metrics summaries keep quantiles for 10s, so a p99 scraped every
30s misses most observations. The _sum/_count rates count every sample.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* grafana: leave raft panels empty when nothing was observed
Clamping the count rate plotted 0 ms for windows without commits.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* master: let the raft metrics test use its own registry
Registering on stats.Gather made go test -count=2 fail with a duplicate
collector registration.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* master: register raft metrics before opening the raft stores
raft.NewRaft starts its run loop before returning, so registering the
sink afterwards could drop the first state and boltdb samples.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* s3api: look up a trailing-slash prefix's directory instead of listing its parent
A prefix such as "account-0001/" names one directory, but the walk found it by
listing the parent with the name as a prefix. SQL stores answer that with a LIKE
that bounds the scan from below only, so in a wide bucket root the query reads
every sibling after the name: seconds for a prefix holding a few keys.
Look the directory up by name and serve it as the level's only entry. A miss,
an error or a non-directory falls back to the prefix listing.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3api: track failover receives atomically
The failover wrapper flags a received reply through a plain bool, which a
callback reading from the filer on several goroutines would race on. Use an
atomic.Bool so the callback may share its filer client.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3api: read directories ahead of a recursive listing walk
A recursive ListObjects walk enters directories one at a time, so a bucket
made of many directories holding one or two keys each costs one filer
round trip per directory, in series: seconds per 1000-key page on a store
whose reads wait on disk.
Read the first window of the directories the walk will enter on a few
workers, nearest first in walk order, and hand each window over when the
walk gets there. Every window the walk reads itself is drained first so its
subdirectories are queued before the walk descends into the first one. The
number of windows held ahead grows while directories end inside their first
window and falls back when one does not, so a page ending inside a large
directory leaves few reads wasted.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3api: drop the prefetcher's record of every directory it was handed
A page can cross any number of empty directories without spending its key
budget, so a set of every announced directory grows with the tree. A
directory is already handled when it is at or behind the walk, held or in
flight, or queued; check those, which the queue and window caps bound.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3api: end a trailing-slash prefix listing at the file its lookup finds
A file named by the prefix lists nothing under it, the same as the
sibling listing it fell back to, which threw the lookup's answer away
and paid the parent scan anyway.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3api: read a walk's own listing ahead instead of draining it
Draining the window before the walk took its first entry made a page
wait for the end of the listing, which a SQL store answering a name
prefix reaches only after scanning the rest of the directory; a page
that fills before then used to return at once. A background reader now
hands the window's directories to the prefetcher as they arrive while
the walk takes each entry, holding at most 1 MiB of entries ahead.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3api: bound the bytes a prefetched listing window holds
Entries with long chunk lists or inline content made the up to 64
held windows of 32 entries an unbounded amount of memory per listing.
A window now stops at 128 KiB of entries and the walk lists the rest
of that directory from the window's last entry.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3api: test recursive listings whose streams break partway
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* s3: require their own actions to change a bucket's CORS or versioning
PUT/DELETE ?cors and PUT ?versioning were authorized as Write, so an
identity granted only object writes on a bucket could replace its CORS
rules or suspend versioning. They now need PutBucketCors,
DeleteBucketCors and PutBucketVersioning (or Admin), and inline policies
granting s3:PutBucketCors, s3:DeleteBucketCors or s3:PutBucketVersioning
map to those actions instead of Write.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3: test that object Write cannot change bucket CORS or versioning
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* admin: offer the bucket CORS and versioning permissions
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3: test each bucket config action with its own identity
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* s3: commit the delete marker in the owner's finalize transaction
createDeleteMarker wrote the marker with mkFile to the gateway's current
filer and then sent RECOMPUTE_LATEST to the object's ring owner. When the
two differ, the owner scans .versions before peer metadata sync delivers
the marker, re-derives the pointer to the previous version, and the
marker stays non-current on every filer.
routedVersionedFinalize now takes an optional version entry and PUTs it
in the same ObjectTransaction ahead of the recompute, the way PutObject
and CompleteMultipartUpload already do. The copy destination keeps
passing nil: its version file is already written on the owner.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3: test a versioned delete when the gateway's filer is not the owner
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3: assert the delete marker and the pointer flip share one transaction
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* s3api: apply copy bucket-default SSE after the ACL re-auth
CopyObject without its own x-amz-server-side-encryption header takes the
destination bucket's default encryption, which CopyObjectHandler writes
into the request headers. It did so before preparePutObjectACL, which
re-authenticates a request that carries an ACL (x-amz-acl or a grant
header) for s3:PutObjectAcl. verifySignedHeadersCoverage then found the
synthesized x-amz-server-side-encryption* headers outside SignedHeaders
and rejected the copy with SignatureDoesNotMatch.
Move the bucket-default injection after the ACL check, so the
re-authentication sees the request as the client signed it. The
destination object still gets the bucket default encryption.
* s3api: assert the copy's KMS key and stored ciphertext
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Stefan Scharlott <stefans@Stefans-MacBook-Pro.local>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* admin: re-send the master vacuum disable every poll so it survives leader changes
The admin disabled the master's scheduled vacuum only when a vacuum plugin
worker connected or disconnected. The master keeps that flag in memory and it
is not replicated through raft, so after a leader change or a leader restart
the new leader ran its own vacuum next to the plugin worker. When both picked
the same volume the master's commit failed ("compact swap aborted: missing
.cpd/.cpx") and it dropped that replica, leaving the volume under-replicated.
Send the desired state on every poll instead, and log on the master only
when the flag actually changes.
* admin: shorten the vacuum sync comments
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Kevin <kevin@leetify.com>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* filer: hide write controls in the UI on the read-only port
The read-only port accepts only GET and HEAD, but its directory page
offered New Folder, Upload, drag-and-drop, rename and delete, each of
which failed with 405. Requests that come through the read-only handler
now render the page without those controls or the upload script.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* filer: test the directory page controls on each port
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* filer: match the rendered row buttons in the port test
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* weed/server: verify data shards against bitrot sidecar before ec decode (#12422)
* ec.decode: verify data shards against the mounted volume's bitrot protection
The decode looked up only the generation-0 .ecsum beside the data base and
each location's data directory, so it missed a sidecar kept with the index
and ignored the generation the volume actually serves.
Take the protection the volume's per-disk runtimes resolved instead (On, else
Invalid, else Off), and test a corrupt block past the first.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* ec.decode: check for live entries before verifying the shards
A volume whose needles are all deleted gets its shards purged on the
no-live-entries status; a failing checksum must not keep it around.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* seaweed-volume: verify data shards against the bitrot protection before a decode
Mirrors the Go volume server: the decode takes the protection the volume's
per-disk runtimes resolved and refuses a data shard that fails its .ecsum
checksum, after the no-live-entries check.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* test: a decode refuses a data shard that fails its .ecsum checksum
Runs against both volume server implementations: the refused decode keeps
every shard and the sidecar, and the needle stays readable through EC.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* ec.decode: reload the bitrot sidecar before verifying the shards
A rebuild's backfill writes the sidecar without reloading the mounted
runtimes, which then still report no protection; resolve it again across
the EC metadata directories, as a shard mount does.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* seaweed-volume: reload the bitrot sidecar before verifying the shards
Mirrors the Go volume server: the decode job resolves the sidecar again on
every runtime of the volume before choosing its protection.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* test: a decode finds a sidecar written after the shards were mounted
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* test: read through the EC shards after the refused decode
The source volume stayed mounted, so the final read never touched the
shards it is meant to check.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* ec.decode: refuse when a sidecar reload reports another layout
ReloadBitrotSidecar logged and dropped the layout-mismatch error, so the
runtime stayed Off and the decode skipped the data-shard check. It now
returns that error and the decode refuses on it.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* seaweed-volume: refuse a decode when a sidecar reload reports another layout
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* test: a decode refuses a late sidecar that records another layout
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* ec.decode: refuse when an unmount left no runtime to resolve the sidecar
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* seaweed-volume: refuse a decode when an unmount left no runtime to resolve the sidecar
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* weed/command: skip remote bucket delete when bucket is unmounted
* remote_storage: count deleting an already-missing bucket as done
filer.remote.gateway can see the same bucket delete more than once, for
example once per filer when several filers peer their metadata. A copy
that still finds the mount mapping calls DeleteBucket again, and S3
(NoSuchBucket), Azure (ContainerNotFound, ContainerBeingDeleted) and GCS
(404) all failed it, which held the sync offset on the delete. A
missing bucket now counts as deleted, the way deleting a missing object
already does.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* filer.remote.gateway: keep the mapping table writable after mount.mapping is deleted
Deleting mount.mapping replaced the gateway's mappings with a table whose
map was nil, so the next bucket create panicked writing its mount into
it. Reset to an empty map instead.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* filer.remote.gateway: count a delete of an absent object as done
filer.remote.sync already treats ErrRemoteObjectNotFound from DeleteFile
as deleted, since GCS reports a missing object that way where S3 and
Azure succeed. The gateway returned it, so a delete event seen twice, as
it is when several filers peer their metadata, held the sync offset on
GCS. Use the same deleteRemoteFile.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* filer.remote.gateway: drop cached remote clients after the new gateway tests
GetRemoteStorage caches clients by configuration, so a repeated run
(-count=2) got the first run's recorder back and saw no calls.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* filer.remote.gateway: keep the default storage across mount.mapping changes
The default storage for new buckets is derived at startup and kept only
in the gateway's copy of the mappings; mount.mapping never stores it.
Replacing or deleting that file dropped it, so a later bucket create
with no -createBucketAt skipped the remote bucket.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* admin: make the cluster lock's fairness window configurable
Scheduled jobs hold the cluster admin lock for their whole run. Once the
lock had been held for a minute, every new job waited until all running
jobs had released it, so with hour-long EC jobs dispatch ran in waves and
most workers sat idle between them.
-adminLock.fairnessWindow keeps the one-minute default; 0 lets new jobs
join the shared hold, and shell clients get the lock when the scheduler
next drains.
Claude-Session: https://claude.ai/code/session_011UpvafcfUuC2mLxNAXLQJi
* admin: wait for the blocked acquire before disabling the fairness window
The wake-up test slept 50 ms before SetFairnessWindow(0), so a late
acquire that started after the change joined without exercising the
Broadcast. Wait until it is registered as a waiter instead.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* weed/shell: resolve collection from topology when ec.decode is given only volumeId
When ec.decode is invoked with -volumeId for a volume that belongs to a named collection without explicitly specifying -collection, DoEcDecode was called with an empty collection string. Target volume servers then rejected VolumeEcShardsCopy with 'CopyFile not found ec volume id <vid>' because they looked under the default collection.
This commit introduces FindEcVolumeCollection to discover the collection name directly from topology metadata (checking EC shard info and regular volume info), ensuring DoEcDecode receives the accurate collection.
* ec.decode: resolve every batch volume's collection from the topology
The batch passed -collection through unless it looked like a regex, so a
comma list (a,b), a {n} quantifier, ALL or _default_ reached the volume
servers as a collection name and the copy found no shards. The selector
only picks the volumes; each one is decoded in its own collection.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* ec.decode: take the collection only from the volume's ec shards
The lookup returned the first record per disk, so a regular volume on an
earlier node answered before the ec shards on a later one. A decode needs
the shards anyway, so their records are the only ones to read.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* ec.decode: leave the -volumeId lookup to DoEcDecode
DoEcDecode already resolves an empty collection from the topology.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* filer.remote.sync: stop a touched file from being restamped forever
A metadata-only update that moves the mtime past RemoteMtime (touch, or
an O_TRUNC open of an empty lock file) goes through UpdateFileMetadata,
which changes nothing on the remote, and the entry is stamped with the
same RemoteEntry and a new LastLocalSyncTsNs. That stamp is an update
event of its own. shouldSendToRemote only looks at LastLocalSyncTsNs
when RemoteMtime and the mtime share a second, so it still reads the
entry as dirty, and the event is stamped again. Nothing is uploaded,
but the daemon keeps rewriting the entry as fast as it reads its own
events back.
Skip the restamp when the backend left the object alone and the event's
entry already carries a stamp taken after its mtime. The touch itself is
still stamped once.
* filer.remote.sync: log the skipped restamp
The other skips in processUpdateEvent log at V(2); this one did not, so
with -v=2 a touched file showed an "update meta" line with nothing after
it.
* filer.remote.sync: skip the restamp of a stamp's own event, whatever the clocks
The skip compared the entry's stamp with its mtime, so an mtime ahead of
the sync's clock (a skewed writer, touch -d) stayed past every stamp and
kept the loop going until the clock caught up. An event that changed only
the RemoteEntry is the sync's own stamp, which needs no other.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* filer.remote.sync: ignore the filer's commit timestamp when spotting a stamp
The filer gives every stored update a new CommitTsNs, so a stamp's event
never compared equal outside the RemoteEntry and a touched file kept being
restamped against a real filer. The test now assigns one per update.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* volume: bound replica forwards so a silent peer cannot hold a write
The Rust volume server forwards each replicated write to its peers with the
shared outgoing reqwest client, which had no connect or request timeout.
When a peer is silently partitioned (packets dropped, no reset) the forward
waits until TCP gives up, minutes later, and the write waits with it. A
chaos test (replication 001, one volume server partitioned) saw every write
on a volume with a copy there block.
- Replica forwards go through replica_request, bounded at
REPLICA_REQUEST_TIMEOUT = 10 s from connect to the response.
- Both outgoing clients start from outgoing_client_builder, which adds a
5 s connect timeout, so new connections to a peer that never answers
fail fast. The shared client gets no overall request timeout, since it
also serves lookups and other transfers.
Tests: a replica that receives the forward and never answers fails the
whole replicated write at the bound, naming that replica; a silent peer
fails the request builder at the bound on a paused clock; a connect that
stalls in the resolver fails at the 5 s connect bound.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* volume: give a replica forward more time for a larger body
The 10 s replica forward bound runs from connect to the end of the
response, so it also covers sending the body. A write near the 256 MB file
size limit needs about 26 MB/s to reach its replica in 10 s, so on a link
that is slower or busy the forward failed and the write returned 500 even
though the peer was answering.
A forward now gets REPLICA_REQUEST_TIMEOUT plus one second per MiB of
body. A bodiless forward keeps the 10 s bound (a 1 MiB write gets 11 s).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* volume: bound each Go replica forward like the Rust one
The Go volume server forwards a replicated write with a 10 s dial bound and
the transport's 30 s response header timeout. A replica that takes the
forward and never answers held the write for 30 s, and a body larger than
the socket buffers to a silently partitioned peer was held until TCP gave
up, since nothing bounds sending the body.
Each forward now runs under replicaRequestBound: 10 s plus one second per
MiB of body, the same bound the Rust volume server puts on its forwards.
Test: a replica that receives the forward and never answers fails the
write at the bound (300 ms in the test); without it the write fails at
30 s on the response header timeout.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* volume: keep proxy variables out of the stalled-connect test
reqwest reads HTTP_PROXY and ALL_PROXY by default, so with a proxy set the
test's request went to the proxy instead of the stalled resolver.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* volume: count a replica forward that hit its bound as a timeout
DistributedOperationResult joined the replica errors into a plain string,
and the uploader reports a cancelled request without wrapping the context
error, so a forward cut off at replicaRequestBound was counted under
server_error in the replication failure metric, as was a response header
timeout before it.
The joined error now unwraps to each replica's error, and a forward that
ran out of its bound reports context.DeadlineExceeded.
Test: the silent replica write raises the timeout failure count by one;
with the flattened join it stays put.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* volume: test that a large Go replica forward gets its body's bound
A replica answers after the bodiless bound but within the bound for a
1 MiB body, so the write fails if replicateTo drops the body scaling.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* volume: run the Rust whole-path replica test with a 1 MiB body
The forward then has to wait the bound for its body, so the test fails if
do_replicated_request stops passing the body length to the bound.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: pingqiu <pingqiu@gmail.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* s3api: name the check that a completion already committed
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3api: answer success when a failed completion already committed
A finalize can fail after its commit: the filer stored the object and the
call still reported an error. The client then got a 500 with an unknown
outcome. When the failure may have left this attempt's object behind,
check the key the way a retry would and answer success with the
committed result when it holds this upload's object.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3api: keep part data when the commit found is another attempt's
The object carrying this upload ID can be a concurrent attempt's commit of
the same upload, and its part selection can include entries this attempt
counts as unused. Free those parts' data only when the committed chunks
are this attempt's own; otherwise drop their metadata only.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3api: test the committed-completion read on versioned buckets
Cover the latest-version read (version id and checksum in the answer),
a suspended bucket whose delete marker is still current, and that the
upload directory is removed without its data.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3api: test versioned recovery through the completion handler
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* seaweed-volume: bound each FetchAndWriteNeedle replica write
A replica that accepted the connection and then went silent held the
request until TCP gave up, because the shared outgoing client has no
request timeout. Each replica write now fails after 30 s, from connect
to the response status, matching the Go volume server.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* volume: bound each FetchAndWriteNeedle replica write
The default replica path uploads through the guarded peer client, which
bounds only the dial and has no response header timeout, and the filer
calls FetchAndWriteNeedle with no deadline. A replica that accepted the
connection and went silent held the request for minutes. Each replica
write, retries included, now has 30 s.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* seaweed-volume: require the silent-peer test to fail at the deadline
A connection error that came back early also passed, since the test
only checked an upper bound. It now also requires the paused clock to
have reached FETCH_AND_WRITE_REPLICA_TIMEOUT.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* seaweed-volume: grow the FetchAndWriteNeedle replica bound with the chunk
The reqwest timeout includes sending the body, and a remote-cache chunk
is a thousandth of the object once it passes the 1000-chunk cap, so a
fixed 30 s would fail a large chunk on a link under ~35 MB/s. The bound
is now 30 s plus 1 s per MiB.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* volume: grow the FetchAndWriteNeedle replica bound with the chunk
The context deadline includes sending the body, so a fixed 30 s would
fail a large remote-cache chunk on a slow link. The bound is now 30 s
plus 1 s per MiB, matching the Rust volume server.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* volume: redial a master that just dropped a registered stream after a second
A registered volume server that loses its master with no peer naming a
leader waited a full pulse before redialing. A restarted master is
usually back within a second, so its volumes stayed unregistered, and
uncached lookups kept failing, for most of a pulse.
For one pulse after a stream a leader had answered ends, redial on the
one-second cap. A master still down after that is redialed at the pulse
pace, so redials never come later than before.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* seaweed-volume: redial a master that just dropped a registered stream after a second
Mirror of the Go heartbeat change. The loop keeps when the last stream a
leader had answered ended, which also stands in for the
confirmed-leader-once flag (None until first registration), and redials
on the one-second cap for a pulse after that.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* seaweed-volume: test that a dropped registered stream records the loss
A fake master confirms the stream and then ends it. The server starts as
registered long ago, so only a recorded loss makes the redial come after
a second instead of a pulse.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* volume: record a lost stream only after its receiver has exited
doHeartbeatWithRetry can return on a send error while the receiver is
still handling the leader's first reply, so the deferred recorder could
read the registered flag too early and keep an old loss time. Cancel the
stream and wait for the receiver before reading it.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* test: require the prompt heartbeat redial to wait out its second
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* seaweed-volume: fail the down master's dials at once in the redial test
Windows takes about two seconds to refuse a connection to a closed
loopback port, so the stream dial and the leader lookup each held the
prompt case past its two-second bound. A listener that closes every
connection fails both dials immediately on every platform.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* ec.rebuild: rebuild shards that fail a checksum scrub, also in complete volumes
ec.rebuild skipped every volume with all shards present, so a shard whose
bytes no longer match the .ecsum sidecar stayed in place until some other
shard went missing. With -checksum, ec.rebuild first scrubs each node's
shards in checksum mode and takes the failing copies out of the plan; a
shard left without a good copy counts as missing and is regenerated.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* ec.rebuild: remove the copies that failed the checksum once a good copy serves
A shard regenerated on the rebuilder left the failing copy mounted on its
old holder, so the volume kept a corrupt duplicate that a later dedup could
keep instead of the good one. Remove each failing copy once the rebuilt
shard is mounted, and right away when another holder already serves a copy
that passed.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* ec.rebuild: test the checksum scrub against fake volume servers
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* ec.rebuild: check the lock before removing a corrupt copy
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* ec.rebuild: print the checksum scrub details
A malformed sidecar or a wholesale mismatch comes back only in the scrub
details, with no broken shard, so the volume would look verified.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* ec.rebuild: fail the run when a corrupt duplicate cannot be removed
For a volume whose only problem is a corrupt duplicate, the removal is the
whole repair, so its failure has to reach the caller.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* ec.rebuild: test duplicate removal failures and a lost lock
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* ec.rebuild: fail the run when a node's checksum scrub fails
The node's copies stay in the plan unverified, so a corrupt shard there is
left in place; the rest of the cluster is still repaired.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* ec.rebuild: preview the corrupt-copy removals in dry-run
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* ec.rebuild: test a failed checksum scrub
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* seaweed-volume: stage optional EC sidecar copies before replacing
VolumeEcShardsCopy opened the destination .vif/.ecsum with File::create
and then streamed into it. A source without the file left a 0-byte .vif,
wiped an existing .ecsum (turning bitrot protection off on that holder),
and a failed stream removed the destination's existing copy.
Stream optional copies into <file>.copying and rename into place only
once the source sent bytes, as the Go volume server's writeToFile does
for ignore_source_file_not_found.
Claude-Session: https://claude.ai/code/session_011UpvafcfUuC2mLxNAXLQJi
* seaweed-volume: keep the drain doc comment on drain_copy_stream_to_file
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* seaweed-volume: stage the .vif copied during missing-index recovery
fetch_ec_index_from_one_peer drained the optional .vif straight into the
local file, so a peer without one left a 0-byte .vif (losing a local EC
ratio) and a failed stream left a partial one. Stage it and rename only once
the peer sent bytes, as Go's recovery does through writeToFile.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* seaweed-volume: test optional copies against a stream that fails mid-transfer
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* seaweed-volume: remove a staged optional copy when the RPC is cancelled
A cancelled CopyFile handler drops its future at the stream await, so the
cleanup after it never ran and the .copying file stayed behind. Go's
writeToFile runs to completion and removes it. A guard now owns the
staging cleanup and is disarmed only once the rename commits.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* seaweed-volume: test optional copies dropped mid-stream
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* seaweed-volume: cancel the staged-copy test only once the staging file exists
tokio's File::create runs on the blocking pool, so a fixed timeout could
drop the .ecsum copy before .copying existed and pass without exercising
the cleanup guard.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* seaweed-volume: run the preallocating max-volume test on a fake disk
The test sized the volume limit from the real free space but the store
subtracts the 1% floor first, so on a disk more than 95% full the second
slot no longer fit and the count came out 1.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* seaweed-volume: start the preallocating max-volume test above the expected count
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
A query's lease tag is deleted by a background task after the query
answers, so the encryption helper could list it and then get a 404 on
HEAD. Skip only that case; any other missing object still fails.
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* seaweed-vector: build and extend the vector index
weed_lance_worker::vector_index builds an IVF_FLAT index in the metric the
table was created with, sizes partitions from the row count, and extends an
index with a new segment or a retrain; the core crate re-exports it and adds
the probes a query spends. Tests cover coverage after build, append, retrain
and compaction.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* weed-vector-cli: index build and index optimize
Logs go to stderr so stdout stays the JSON a caller parses.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* test/s3vectors: PERF_INDEX builds the index before the timed queries
The cluster fixture hands out its catalog URL, the Makefile builds
weed-vector-cli, and the perf suite builds the index after ingest and
reports the build time and segment count.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* seaweed-vector: probe a tenth of the index's partitions, more under a filter
A query that names no probe count gets one from the index's partition
count. Lance searches at least that many partitions, and under a prefilter
keeps going until it has k rows; without one it stops there.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* weed-vector: --nprobes overrides the probe policy
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* seaweed-vector: test that a build takes the table's metric
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* test/s3vectors: make clean removes weed-vector-cli
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* seaweed-vector: measure recall with vectors the table does not hold
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* weed-vector: keep the read cache's doc comment on with_read_cache
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* weed-worker: lance_vector_index builds the index of an S3 Vectors table past a row threshold
Detection keeps the tables a vector bucket created, with at least min_rows
rows (50,000 by default) and no index on their vector column. Execution
builds the IVF_FLAT index in the table's metric with sqrt(rows) partitions.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* seaweed-vector: optimize only the indices on the vector column
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* seaweed-vector: resolve the query metric with index::distance_metric
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* seaweed-vector: test the L2 metric under the name CreateIndex stores
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* test/s3vectors: build the index through the admin server and the Lance worker
VECTOR_WORKER=1 adds weed admin and weed-worker to the cluster. The new
test sets the job's threshold, runs it through the plugin API, checks
coverage and recall, and that a row written after the build is found.
CI gains the worker leg for both volume servers.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* weed-worker: read an index's partitions, and when a table has outgrown them
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* seaweed-vector: probes from the shared partition reader, partitions in coverage
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* weed-worker: a vector index reindexes at a tenth of its rows and retrains once the table outgrows it
lance_optimize_indices proposes a vector table once its uncovered rows pass
min(the absolute budget, a tenth of the indexed rows), and trains the index
again instead of extending it when sqrt(rows) has reached twice its
partitions. The end-to-end test covers both.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* weed-worker: open a table inside a caller's Lance session
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* weed-vector: keep opened indices between queries in one session
Every open used a fresh Lance session, so the index and metadata caches
died with the request and each query loaded the IVF index again.
--index-cache-mb sizes the shared cache, 512 MiB by default.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* test/s3vectors: PERF_CLUSTERS draws the perf suite's vectors around shared centres
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* test/s3vectors: the index test draws embedding-like vectors
Independent Gaussian vectors have no locality for an IVF index to use, so
recall through one only tracks the fraction of partitions probed; points
around shared centres behave like embeddings. The test must also run
before the one that stops the engine.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* test/s3vectors: the optimize and retrain steps draw embedding-like vectors
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* seaweed-vector: test that compaction drops the deleted key from tied hits
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* seaweed-vector: test that an append-only optimize serves the new row
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* test/s3vectors: record PERF_NOISE in the perf report config
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* seaweed-vector: a filtered query widens to four times the probes, not without bound
With no upper bound Lance prepares every partition for the late search
even when the minimum probes already hold k rows, and a filtered query
through the index cost as much as the exact scan. A query that names its
own probe count gets exactly that count.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* test/s3vectors: report recall through an index without asserting it
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* test/s3vectors: count a hit tying the k-th best score toward recall
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* admin: lance_vector_index runs in the Lance lane
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* weed-worker: check a table's index before counting its rows
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* test/s3vectors: make build and clean cover the worker leg's binaries
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* weed-worker: a retrain rebuilds the index with partitions for the row count
Lance's OptimizeOptions::retrain does not size the partitions: it left a
6,000-row index with one. The retrain goes through the builder with
sqrt(rows) partitions, for the job and for weed-vector-cli index optimize
--retrain alike. The end-to-end test no longer expects a merge to keep the
partition count.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* test/s3vectors: allow float32 rounding at the recall cutoff
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* test/s3vectors: count a repeated hit key once toward recall
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* test/s3vectors: credit cutoff ties only up to the slots the top-k leaves
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* test/s3vectors: assert every filtered hit passes the filter, indexed or not
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* weed-worker: coverage over an index's names, one name extended, a merge at the delta limit
A sharded build leaves one name per shard, each missing the others'
fragments, so coverage is the union of every index on the column and an
optimize gives new rows to the name with the fewest deltas only. At
MAX_DELTAS the policy passes OptimizeOptions::merge(deltas); Lance's
default options may append instead of merging. A retrain keeps an index's
own metric and rebuilds only IVF_FLAT indices.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* weed-worker: the optimize job measures and extends a vector table's indices together
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* seaweed-vector: tests for bounded deltas, disjoint shards and a retrain's metric
Nine append-and-optimize cycles give delta counts 2 to 8 and then 1; two
names on disjoint fragments stay disjoint through an optimize; a cosine
index on a table without metric metadata is still cosine after a retrain.
The end-to-end test expects the appended delta after the first optimize.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* weed-vector: forget the shared session when a table was dropped and created again
The session keys index lists and transactions by dataset version, and a
table created again at the same location restarts its versions. An open
that finds an older version than the newest seen at that URI, or the same
version committed later, replaces the session and opens again.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* weed-vector: test an open after another engine created the table again
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* weed-worker: retrain only a single IVF_FLAT vector index name
The job compared each shard name's partitions with the whole table's
rows, so a sharded index was always due, and a non-IVF_FLAT index made
optimize bail on every run before the new rows were indexed.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* weed-worker: refuse to retrain a vector index sharded over several names
A rebuild per name over the whole table left every shard covering every
fragment, so a query searched each row once per shard.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* seaweed-vector: test which vector indices a retrain is due for
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* weed-vector: replace the session in the same lock that records a recreated table
A second open of the recreated table could see the new version recorded
and the old session still shared, and keep the dropped table's index list.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* weed-worker: skip a vector table whose rows cannot be counted
Detection returned on the first table whose count_rows failed, so one
unreadable table kept every other table from being proposed. It now skips
that table like an open or index failure.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* grafana: panel for vector indexes the Lance worker built
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* seaweed-vector: size default probes from the largest index on the column
A sharded build gives the column one index name per shard. The first
name alone could be a small shard, and its probe count would then be
spent on a larger one.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* seaweed-vector: skip the probe lookup when a query is given no segments
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* weed-vector: open every table in its own Lance session again
One session shared by every open keeps a table's index loaded between
queries, but Lance keys what it caches by location and version, so a
table dropped and created again at the same location could be served
the old table's index list. Telling the two apart from version and
commit time needs a map that only grows, a retry loop and clocks that
agree across engines. A private session per open, as before, has no
such state. Measured at 50k x 384 through an index, eight clients: 164
queries/s with the shared session, 137 without; one client is unchanged.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* weed-vector: reject --nprobes 0
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* seaweed-vector: read the partition count from the opened index
Lance's index statistics took 3.6 ms a query on a 200k-row table with
447 partitions. Opening the index and asking it takes 0.4 ms, and the
scan that follows finds the index already open.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* ci: give the S3 Vectors worker legs the vendored protoc
The worker workspace has no protoc of its own for lance's build
scripts, so the worker legs failed at their first build.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* weed-worker: give a vector index build an hour before the admin times it out
The job set no execution timeout, so the admin's default of 90 seconds
applied, and a build that trains on a large table would be reported as
failed and retried while it was still running.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* weed-worker: lance_vector_index proposes at most the admin's max_results
Every qualifying table went into one message however many there were.
The detector stops at the limit the request carries and sets has_more,
as the Go detectors do.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* weed-worker: a vector index job re-checks the table it was proposed for
A table emptied since detection completes without a build, and one
without a vector column fails instead of reporting it already indexed.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* weed-worker: lance_optimize_indices gets a build's execution timeout
A retrain rebuilds the index over the whole table, which the admin's
default of 90 seconds would cut off on a large one.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* weed-worker: a vector index run window as long as one build
An attempt is capped at the end of the job type's run window plus the
drain grace, and the default window is 30 minutes, so a build could be
cut off before its own hour was up. The window is an hour too, as the
Go iceberg handler sets both.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* weed-worker: a lance_optimize_indices run window as long as one retrain
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* weed-worker: vector index proposals declare a build's runtime
A run started by hand while the job type's schedule is off takes no
timeout from the descriptor, only the admin's 90 seconds. The admin
gives a job at least the runtime its proposal declares, so the build
and the optimize proposals of a vector table declare the hour.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* seaweed-vector: coverage reports no partitions for an index of unknown coverage
Reading the partition count goes through Lance's index statistics,
which fail on an index that records no fragment bitmap, and the whole
coverage report failed with them.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* weed-worker: an index that records no fragments is not read as uncovered
Its rows counted as uncovered, so detection proposed the table and the
execution then failed in Lance's statistics. Coverage fails for such an
index instead, and detection skips the table with a warning.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* seaweed-vector: coverage reads partition counts from the opened index
Lance's index statistics rewrite the manifest of a table whose
fragments carry no row counts, which a coverage report must not do and
cannot do with read-only credentials. The report asks each opened
segment instead, as the probe lookup does, and the statistics-based
reader has no caller left.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
---------
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
* seaweed-vector: build and extend the vector index
weed_lance_worker::vector_index builds an IVF_FLAT index in the metric the
table was created with, sizes partitions from the row count, and extends an
index with a new segment or a retrain; the core crate re-exports it and adds
the probes a query spends. Tests cover coverage after build, append, retrain
and compaction.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* weed-vector-cli: index build and index optimize
Logs go to stderr so stdout stays the JSON a caller parses.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* test/s3vectors: PERF_INDEX builds the index before the timed queries
The cluster fixture hands out its catalog URL, the Makefile builds
weed-vector-cli, and the perf suite builds the index after ingest and
reports the build time and segment count.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* seaweed-vector: probe a tenth of the index's partitions, more under a filter
A query that names no probe count gets one from the index's partition
count. Lance searches at least that many partitions, and under a prefilter
keeps going until it has k rows; without one it stops there.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* weed-vector: --nprobes overrides the probe policy
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* seaweed-vector: test that a build takes the table's metric
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* test/s3vectors: make clean removes weed-vector-cli
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* seaweed-vector: measure recall with vectors the table does not hold
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* weed-vector: keep the read cache's doc comment on with_read_cache
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* weed-worker: lance_vector_index builds the index of an S3 Vectors table past a row threshold
Detection keeps the tables a vector bucket created, with at least min_rows
rows (50,000 by default) and no index on their vector column. Execution
builds the IVF_FLAT index in the table's metric with sqrt(rows) partitions.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* seaweed-vector: optimize only the indices on the vector column
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* seaweed-vector: resolve the query metric with index::distance_metric
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* seaweed-vector: test the L2 metric under the name CreateIndex stores
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* test/s3vectors: build the index through the admin server and the Lance worker
VECTOR_WORKER=1 adds weed admin and weed-worker to the cluster. The new
test sets the job's threshold, runs it through the plugin API, checks
coverage and recall, and that a row written after the build is found.
CI gains the worker leg for both volume servers.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* weed-worker: open a table inside a caller's Lance session
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* weed-vector: keep opened indices between queries in one session
Every open used a fresh Lance session, so the index and metadata caches
died with the request and each query loaded the IVF index again.
--index-cache-mb sizes the shared cache, 512 MiB by default.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* test/s3vectors: PERF_CLUSTERS draws the perf suite's vectors around shared centres
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* test/s3vectors: the index test draws embedding-like vectors
Independent Gaussian vectors have no locality for an IVF index to use, so
recall through one only tracks the fraction of partitions probed; points
around shared centres behave like embeddings. The test must also run
before the one that stops the engine.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* seaweed-vector: test that compaction drops the deleted key from tied hits
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* seaweed-vector: test that an append-only optimize serves the new row
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* test/s3vectors: record PERF_NOISE in the perf report config
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* seaweed-vector: a filtered query widens to four times the probes, not without bound
With no upper bound Lance prepares every partition for the late search
even when the minimum probes already hold k rows, and a filtered query
through the index cost as much as the exact scan. A query that names its
own probe count gets exactly that count.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* test/s3vectors: report recall through an index without asserting it
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* test/s3vectors: count a hit tying the k-th best score toward recall
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* admin: lance_vector_index runs in the Lance lane
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* weed-worker: check a table's index before counting its rows
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* test/s3vectors: make build and clean cover the worker leg's binaries
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* test/s3vectors: allow float32 rounding at the recall cutoff
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* test/s3vectors: count a repeated hit key once toward recall
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* test/s3vectors: credit cutoff ties only up to the slots the top-k leaves
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* test/s3vectors: assert every filtered hit passes the filter, indexed or not
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* weed-vector: forget the shared session when a table was dropped and created again
The session keys index lists and transactions by dataset version, and a
table created again at the same location restarts its versions. An open
that finds an older version than the newest seen at that URI, or the same
version committed later, replaces the session and opens again.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* weed-vector: test an open after another engine created the table again
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* weed-vector: replace the session in the same lock that records a recreated table
A second open of the recreated table could see the new version recorded
and the old session still shared, and keep the dropped table's index list.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* weed-worker: skip a vector table whose rows cannot be counted
Detection returned on the first table whose count_rows failed, so one
unreadable table kept every other table from being proposed. It now skips
that table like an open or index failure.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* grafana: panel for vector indexes the Lance worker built
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* seaweed-vector: size default probes from the largest index on the column
A sharded build gives the column one index name per shard. The first
name alone could be a small shard, and its probe count would then be
spent on a larger one.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* seaweed-vector: skip the probe lookup when a query is given no segments
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* weed-vector: open every table in its own Lance session again
One session shared by every open keeps a table's index loaded between
queries, but Lance keys what it caches by location and version, so a
table dropped and created again at the same location could be served
the old table's index list. Telling the two apart from version and
commit time needs a map that only grows, a retry loop and clocks that
agree across engines. A private session per open, as before, has no
such state. Measured at 50k x 384 through an index, eight clients: 164
queries/s with the shared session, 137 without; one client is unchanged.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* weed-vector: reject --nprobes 0
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* seaweed-vector: read the partition count from the opened index
Lance's index statistics took 3.6 ms a query on a 200k-row table with
447 partitions. Opening the index and asking it takes 0.4 ms, and the
scan that follows finds the index already open.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* ci: give the S3 Vectors worker legs the vendored protoc
The worker workspace has no protoc of its own for lance's build
scripts, so the worker legs failed at their first build.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* weed-worker: give a vector index build an hour before the admin times it out
The job set no execution timeout, so the admin's default of 90 seconds
applied, and a build that trains on a large table would be reported as
failed and retried while it was still running.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* weed-worker: lance_vector_index proposes at most the admin's max_results
Every qualifying table went into one message however many there were.
The detector stops at the limit the request carries and sets has_more,
as the Go detectors do.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* weed-worker: a vector index job re-checks the table it was proposed for
A table emptied since detection completes without a build, and one
without a vector column fails instead of reporting it already indexed.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* weed-worker: a vector index run window as long as one build
An attempt is capped at the end of the job type's run window plus the
drain grace, and the default window is 30 minutes, so a build could be
cut off before its own hour was up. The window is an hour too, as the
Go iceberg handler sets both.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
---------
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
* seaweed-vector: build and extend the vector index
weed_lance_worker::vector_index builds an IVF_FLAT index in the metric the
table was created with, sizes partitions from the row count, and extends an
index with a new segment or a retrain; the core crate re-exports it and adds
the probes a query spends. Tests cover coverage after build, append, retrain
and compaction.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* weed-vector-cli: index build and index optimize
Logs go to stderr so stdout stays the JSON a caller parses.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* test/s3vectors: PERF_INDEX builds the index before the timed queries
The cluster fixture hands out its catalog URL, the Makefile builds
weed-vector-cli, and the perf suite builds the index after ingest and
reports the build time and segment count.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* seaweed-vector: probe a tenth of the index's partitions, more under a filter
A query that names no probe count gets one from the index's partition
count. Lance searches at least that many partitions, and under a prefilter
keeps going until it has k rows; without one it stops there.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* weed-vector: --nprobes overrides the probe policy
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* seaweed-vector: test that a build takes the table's metric
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* test/s3vectors: make clean removes weed-vector-cli
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* seaweed-vector: measure recall with vectors the table does not hold
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* weed-vector: keep the read cache's doc comment on with_read_cache
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* seaweed-vector: optimize only the indices on the vector column
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* seaweed-vector: resolve the query metric with index::distance_metric
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* seaweed-vector: test the L2 metric under the name CreateIndex stores
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* weed-worker: open a table inside a caller's Lance session
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* weed-vector: keep opened indices between queries in one session
Every open used a fresh Lance session, so the index and metadata caches
died with the request and each query loaded the IVF index again.
--index-cache-mb sizes the shared cache, 512 MiB by default.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* test/s3vectors: PERF_CLUSTERS draws the perf suite's vectors around shared centres
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* seaweed-vector: test that compaction drops the deleted key from tied hits
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* seaweed-vector: test that an append-only optimize serves the new row
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* test/s3vectors: record PERF_NOISE in the perf report config
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* seaweed-vector: a filtered query widens to four times the probes, not without bound
With no upper bound Lance prepares every partition for the late search
even when the minimum probes already hold k rows, and a filtered query
through the index cost as much as the exact scan. A query that names its
own probe count gets exactly that count.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* test/s3vectors: report recall through an index without asserting it
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* test/s3vectors: count a hit tying the k-th best score toward recall
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* test/s3vectors: allow float32 rounding at the recall cutoff
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* test/s3vectors: count a repeated hit key once toward recall
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* test/s3vectors: credit cutoff ties only up to the slots the top-k leaves
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* test/s3vectors: assert every filtered hit passes the filter, indexed or not
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* weed-vector: forget the shared session when a table was dropped and created again
The session keys index lists and transactions by dataset version, and a
table created again at the same location restarts its versions. An open
that finds an older version than the newest seen at that URI, or the same
version committed later, replaces the session and opens again.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* weed-vector: test an open after another engine created the table again
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* weed-vector: replace the session in the same lock that records a recreated table
A second open of the recreated table could see the new version recorded
and the old session still shared, and keep the dropped table's index list.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* seaweed-vector: size default probes from the largest index on the column
A sharded build gives the column one index name per shard. The first
name alone could be a small shard, and its probe count would then be
spent on a larger one.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* seaweed-vector: skip the probe lookup when a query is given no segments
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* weed-vector: open every table in its own Lance session again
One session shared by every open keeps a table's index loaded between
queries, but Lance keys what it caches by location and version, so a
table dropped and created again at the same location could be served
the old table's index list. Telling the two apart from version and
commit time needs a map that only grows, a retry loop and clocks that
agree across engines. A private session per open, as before, has no
such state. Measured at 50k x 384 through an index, eight clients: 164
queries/s with the shared session, 137 without; one client is unchanged.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* weed-vector: reject --nprobes 0
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* seaweed-vector: read the partition count from the opened index
Lance's index statistics took 3.6 ms a query on a 200k-row table with
447 partitions. Opening the index and asking it takes 0.4 ms, and the
scan that follows finds the index already open.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
---------
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
* seaweed-vector: build and extend the vector index
weed_lance_worker::vector_index builds an IVF_FLAT index in the metric the
table was created with, sizes partitions from the row count, and extends an
index with a new segment or a retrain; the core crate re-exports it and adds
the probes a query spends. Tests cover coverage after build, append, retrain
and compaction.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* weed-vector-cli: index build and index optimize
Logs go to stderr so stdout stays the JSON a caller parses.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* test/s3vectors: PERF_INDEX builds the index before the timed queries
The cluster fixture hands out its catalog URL, the Makefile builds
weed-vector-cli, and the perf suite builds the index after ingest and
reports the build time and segment count.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* seaweed-vector: test that a build takes the table's metric
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* test/s3vectors: make clean removes weed-vector-cli
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* seaweed-vector: optimize only the indices on the vector column
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* seaweed-vector: resolve the query metric with index::distance_metric
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* seaweed-vector: test the L2 metric under the name CreateIndex stores
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* test/s3vectors: PERF_CLUSTERS draws the perf suite's vectors around shared centres
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* seaweed-vector: test that compaction drops the deleted key from tied hits
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* seaweed-vector: test that an append-only optimize serves the new row
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* test/s3vectors: record PERF_NOISE in the perf report config
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* test/s3vectors: report recall through an index without asserting it
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* test/s3vectors: count a hit tying the k-th best score toward recall
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* test/s3vectors: allow float32 rounding at the recall cutoff
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* test/s3vectors: count a repeated hit key once toward recall
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* test/s3vectors: credit cutoff ties only up to the slots the top-k leaves
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* test/s3vectors: assert every filtered hit passes the filter, indexed or not
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
Both were generated at the repo root, so their embedded source paths read
weed/admin/view/... where every other admin template, generated by
weed/admin's Makefile, reads view/.... No markup changes.
Claude-Session: https://claude.ai/code/session_011UpvafcfUuC2mLxNAXLQJi
* admin/plugin: fail a dispatch its restarted worker no longer lists
A job dispatched to a worker that then restarts dies with the old
process. The worker re-registers under the same id on a new stream and
its heartbeats stop listing the job, but nothing ended the pending
execution: the dispatcher kept waiting and held the executor slot until
the attempt's execution timeout.
Each pending execution now records the stream it went out on. An
itemized heartbeat from a newer stream of that worker that leaves the
job out twice in a row fails it through the expire path, so the
dispatch returns, the slot is released and the retry machinery takes
over. Jobs on the heartbeat's own stream, jobs the new stream still
lists, and heartbeats that count slots without itemizing them are left
alone.
Claude-Session: https://claude.ai/code/session_011UpvafcfUuC2mLxNAXLQJi
* admin/plugin: let a partly itemized listing clear a dispatch's misses
A heartbeat that counted more slots than it itemized returned before the
listed jobs had their misses reset, so an omission, a partial heartbeat
listing the job, and another omission failed a job the worker was still
running.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* admin/plugin: look up the worker's stream before tracking the attempt
A dispatch to a worker with no stream returned after the job was already
recorded as assigned, and that record kept new proposals with the same
dedupe key out until the stale-job sweep expired it.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* admin/plugin: fail the attempt whose request could not be sent
The dispatch now sends on the stream it looked up before tracking the
attempt, so a worker that reconnects in between makes the send fail.
That returned with the job still tracked as assigned, and with the
worker back nothing closed the record for the 24h stale timeout while
its dedupe key kept new proposals out. The attempt is now completed as
failed, after it is unregistered so the completion does not wait on a
channel nobody reads.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* admin/plugin: leave a send failure to the expiry that already claimed it
The send-error path and an orphan expiry could both complete the same
attempt and record two failed runs for it. The send-error path now
completes the attempt only if no expiry has marked it failed, decided
under the same lock that unregisters it.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* admin/plugin: record a send failure whose entry Shutdown already removed
Shutdown closes shutdownCh before it unregisters pending dispatches, so a
blocked send returns while its map entry is gone. Read the expiry claim
from the dispatch's own state instead of the map lookup, so the attempt
is still failed and does not stay assigned.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* admin/plugin: match a padded job id against its trimmed listing
Heartbeat work ids are trimmed before the orphan check, so compare the
dispatch's job id trimmed too; otherwise a job id with surrounding
spaces never matches its own listing and the running job is failed.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* admin/plugin: skip orphan misses once a completion was handed over
The buffer length only shows a completion the dispatcher has not taken
yet; after it takes one, the attempt stays registered until the dispatch
returns. Mark the hand-off under the pending lock and check that flag
instead, so heartbeats in that gap cannot fail a finished attempt.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* seaweed-volume: report EC file and delete counts in heartbeat deltas
The full EC heartbeat already carries file_count and delete_count, but the
incremental new/deleted shard messages built in
collect_ec_shard_delta_messages left both at zero. A volume the master first
learns of through a delta then reads 0 files until the next full EC
heartbeat. Fill both from file_and_delete_count, the same source the full
heartbeat uses, computed once per volume rather than once per shard.
Claude-Session: https://claude.ai/code/session_011UpvafcfUuC2mLxNAXLQJi
* volume: report EC file and delete counts in mount notifications
Go parity with the Rust delta change: the NewEcShardsChan messages sent on
MountEcShards and on startup shard load now carry FileCount and DeleteCount
from EcVolume.FileAndDeleteCount, the same values the full EC heartbeat
reports, instead of leaving them at zero.
Claude-Session: https://claude.ai/code/session_011UpvafcfUuC2mLxNAXLQJi
* seaweed-volume: leave count-only EC changes out of the shard delta
The delta messages now carry file_count and delete_count, so the whole-message
comparison re-sent every shard of a volume after any delete, and the master
broadcast each one as a new EC vid. Compare the shard fields only; the full EC
heartbeat already carries the counts.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* topology: mount messages now carry EC counts, only older servers omit them
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* storage: guard the captured EC mount messages in the disk-type tests
The drain goroutine appended to the slice that waitForShardMsg polled with no
lock, which go test -race reports.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* storage: test EC counts on the startup notification and repeated .ecj ids
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* seaweed-volume: move file_count too in the count-only delta test
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* shell: accept ReadAcp and WriteAcp in per-bucket and service account actions
s3.bucket.access, s3.anonymous.set and s3.serviceaccount.create rejected
ReadAcp and WriteAcp, so the ACL actions the S3 gateway enforces could not
be granted through them. Share one canonical action table across the
three commands and list the ACL actions in their help.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* shell: list ReadAcp and WriteAcp in s3.configure and s3.circuitBreaker help
The -actions usage of both commands stopped at Admin, so the ACL actions
the gateway enforces could only be found in the source. s3.circuitBreaker
now prints the actions it actually accepts.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* admin: offer ReadAcp and WriteAcp in the object store user permissions
The create and edit user forms only offered Read, Write, List, Tagging and
Admin among the bucket permissions, so an identity could not be given the
ACL actions from the UI even though uploads with x-amz-acl need WriteAcp.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3api: log the WriteAcp denial of an upload that sets an ACL
An upload or copy that carries x-amz-acl or an x-amz-grant-* header also
needs WriteAcp. When that check denies, the client only sees a bare
AccessDenied and nothing reached the log at the default verbosity, so
the header was never suspected. Log the identity and the missing action.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3api: quote the key in the ACL upload denial log
An object key can carry a decoded newline, which let a caller add forged
lines to the default log. Quote the bucket and key, and word the message
so it also fits a policy deny, not only a missing grant.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* shell: say ACL uploads need both Write and WriteAcp
The s3.configure example granted WriteAcp alone, which still fails the
upload's Write check.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* shell: parse S3 action lists with one shared helper
s3.bucket.access, s3.anonymous.set and s3.serviceaccount.create each had
their own validation loop, so a test of the table did not cover the
parsing the commands run. They now call canonicalActionList, which skips
empty entries and repeats. s3.anonymous.set validates before contacting
IAM, and s3.bucket.access still rejects a list with no actions.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* admin: keep bucket-scoped actions the user form cannot show
When a user has both global and bucket-scoped actions, the edit form
shows only the global ones, and saving replaced the identity's actions
with what the form rebuilt, deleting the scoped grants. Carry those
actions through the save.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* admin: list hidden bucket grants in the user edit form
Restoring the bucket-scoped grants the form cannot show kept them active
even when an admin narrowed the user's permissions, with no way to see
or drop them. Show them as checked boxes under Other Bucket Permissions;
unchecking one removes it on save.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* shell: reject an s3.anonymous.set -access list with no actions
A blank list such as ",," parsed to nothing and cleared the bucket's
anonymous grants. Use "none" for that, as s3.bucket.access does.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3api: skip the ACL upload denial log for anonymous requests
Anonymous uploads with ACL headers can be sent by anyone, so logging
each denial at the default level lets unauthenticated clients flood the
log. Keep the line for named identities.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* admin: count kept bucket grants when checking an edited user for permissions
The empty check ran before the checked Other Bucket Permissions were
merged, so a user left with only those grants could not be saved.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* admin: list bucket grants of Admin users in the user edit form
parseBucketPermissions returned before reading scoped actions when the
user had Admin, so removing Admin and saving dropped grants such as
WriteAcp:bucket.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* admin: keep unpaired bucket grants as separate checkboxes in the user edit form
The form saves every selected permission on every selected bucket, so
grants such as ReadAcp:a with WriteAcp:b came back as all four pairs.
Show the grants through the picker only when they cover that full
product; otherwise list each grant on its own. Opening the form also
resets the bucket scope left over from the previous user.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* admin: keep grants the edit pickers have no option for
A permission x bucket product was shown through the pickers even when a
bucket was not in the bucket list (deleted, prefix or wildcard grants)
or a permission had no picker option (DeleteBucket, PutBucketQuota).
Saving then dropped those grants. Such grants now go to the checkbox
list, global ones included, and are submitted unchanged.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* shell: name the s3.bucket.access anonymous example by what it grants
Read,Write,List leaves out ReadAcp and WriteAcp, so it is not full
access.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* admin: keep unlisted global grants of Admin users in the user edit form
parseBucketPermissions returned for Admin before splitting global grants
into picker options and the rest, and only raw option values were
re-selected. Saving an Admin user dropped s3tables:* and grants with no
picker option such as DeleteBucket. Admin users now use the same split.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* admin: keep the /* suffix of bucket grants in the user edit form
parseBucketPermissions dropped a trailing /* before matching bucket
options, so saving rewrote ReadAcp:bucket/* as ReadAcp:bucket. The bare
form also matches bucket-level requests such as GET ?acl, so the save
widened the grant. A /* bucket is not a picker option, so the grant now
stays in the checkbox list as written.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* shell: test that s3.bucket.access saves ReadAcp and WriteAcp grants
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
A Content-Type form field such as "\n" was copied verbatim into the
multipart part header sent to the volume server, which broke the
framing and surfaced as a Content-MD5 mismatch and a 500. Form fields
promoted to request headers must be valid header values, as on a PUT;
otherwise answer 400 MalformedPOSTRequest.
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* fix(s3): accept fresh admin tokens when context carries stale authorization (#12390)
Signed-off-by: Mohd Quamar Tyagi <104281681+Tyagiquamar@users.noreply.github.com>
* security: share the admin Bearer check between S3 and admin-plane peers
Move the authorization scan into security.CheckGrpcAdminToken, built on
IsFilerAdminToken, and call it from CheckGrpcAdminPeer and the S3 IAM cache
check. Both now accept a valid admin token in any authorization value
instead of S3 re-implementing the decode and claim checks.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* filer: accept a fresh admin token after a stale one on IAM gRPC
The S3 IAM cache check is meant to mirror the filer's, so route the filer's
checkAdminAuth through security.CheckGrpcAdminToken too. IamGrpcStore appends
its token to the caller's context, so a forwarded older token would otherwise
still shadow it here.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* security: parse at most four authorization values per admin call
Scanning every value let one request repeat a junk Bearer token (cheap over
HPACK) tens of thousands of times and burn about half a second of JWT parsing
on an unauthenticated port. Real callers send their own token plus at most one
forwarded from their context.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* credential: match the other WithS3InternalAdminAuth replace fix
The plugin worker fix makes the same set-instead-of-append change to
WithS3InternalAdminAuth with different code and a different test at the same
spot, so whichever lands second would conflict. Use the identical code and
test so the two merge cleanly in either order.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* security: test an expired admin token ahead of a fresh one
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* security: test that a same-key session token is not an admin token
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* filer: test IAM gRPC with only invalid authorization values
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Signed-off-by: Mohd Quamar Tyagi <104281681+Tyagiquamar@users.noreply.github.com>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* storage: search corrupt shard sets when an EC needle fails to rebuild
A needle whose reassembled bytes fail to parse is rebuilt with its own
shards excluded. When other shards are corrupt over the same range, that
decode reads them as inputs, and the retry dropped only one input at a
time, so two or more extra corrupt shards left the needle unreadable even
within parity.
After the single pass fails, try every set of up to ParityShards shards
that holds a suspect as the corrupt set, smallest first. Suspect ranges
inside the set are rebuilt from the shards outside it and the rest keep
their bytes, so a clean suspect can still feed the decode. Sets whose
decodes read the same inputs are tried once, and the search is bounded
by set count and by rebuilt bytes past the single-shard sets.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* seaweed-volume: search corrupt shard sets when an EC needle fails to rebuild
Mirror the Go read path: after the single pass over the suspects fails,
gather each interval from every other shard once and try each set of up
to parity_shards shards holding a suspect as the corrupt set, smallest
first, with the same dedup and bounds. The search runs on the blocking
pool, holding the recovery permit, since it can take many decodes.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* ec: charge only multi-shard sets against the rebuild byte budget
The budget check already let single-shard sets through, but each one
was still charged, so a needle spanning many shards could spend the
budget before the first multi-shard set.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* ec: bound the corrupt-set gather to the recovery budget
The search gathered every other shard per interval, so a large needle
spanning several shards held up to Total-1 copies of itself under a
permit capped at the budget. Gather only as many as the budget covers,
but never fewer than the suspects-excluded gather the fallback always
held, so single-suspect recovery keeps every other shard.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* ec: gather at least one decode's worth for the corrupt-set search
A needle over more shards than parity skips the single pass, and its
suspects-excluded width can fall below DataShards, so the bounded
gather left even one corrupt shard unrecoverable. Floor the width at
DataShards, which the per-suspect rebuild this search replaced read.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* ec: rebuild one suspect at a time when the search gather is narrow
Flooring the search gather at DataShards made a large needle over more
shards than parity hold ten copies of itself, where the per-suspect
rebuild it replaced held one interval's fan-out. Keep that rebuild
after the search, and skip the search when the budget leaves fewer
inputs than one decode. Parity slack still widens the gather to every
shard outside the suspects, as the dropped-input retry did.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* ec: keep the dropped-input retry and run the set search within budget
A narrowed search gather skipped only each interval's own shard, so it
could read corrupt suspects in place of healthy shards the old retry
kept, and its byte bound could stop before that retry's hypotheses.
Restore the dropped-input retry as it was, move the search into its
own function after it, and run the search only when every interval's
full gather fits the recovery budget. Past the budget a read takes the
same paths, with the same footprint, as before the search.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* grafana: chart EC intervals served by reconstruction
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* grafana: chart volume server storage IO errors and quarantine
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* grafana: chart frozen volume growth and duplicate volume ids
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* grafana: chart filer metadata log gaps and replay failures
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* grafana: add failed bytes to filer sync throughput
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* grafana: chart remote mount cache reads and evictions
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* grafana: chart plugin worker detection duration
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* grafana: show duplicate volume ids whose first detection was the only one
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* grafana: keep cluster apart in the master growth and duplicate id panels
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* volume: count EC reads that fall back to reconstruction
When a needle read can't read an EC shard interval directly (the shard isn't
mounted here and no holder answers), both volume servers rebuild it from the
other shards and the read still succeeds, with nothing to show for it.
Count those fallbacks in SeaweedFS_volumeServer_ec_degraded_read_total{result}
(success, failure), one per interval, at the one place each server falls back:
Go readOneEcShardInterval, Rust fetch_one_interval. Corruption repair and scrub
aren't counted. A line binding each one to the needle and the volume's encode
generation is logged at V(3) in Go and debug in Rust.
* volume: name the EC counter for what it counts
Reed-Solomon reconstruction can't tell corrupt inputs from good ones, and the
needle is validated later, so "success" only ever meant the reconstruction
produced data. Rename the counter to
SeaweedFS_volumeServer_ec_reconstructed_interval_total and the log line to
"ec reconstructed interval", say so in the help text, and pin the behaviour
with a test: an interval rebuilt from corrupt parity counts success while the
read itself fails.
* storage: restore stderr when the log-line test stops early
A fixture failure in readEcNeedle ends the test through t.Fatalf, which
skipped restoring os.Stderr and closing the pipe, so later tests wrote
into a pipe nobody read.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* s3tables: validate policy conditions and fail safe on unusable ones
* s3vectors: reject a condition operator with no keys
The shared evaluator now treats a keyless operator as unusable, so a
stored keyless Allow would never grant while the write succeeded.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3vectors: reject an Allow whose Condition is null
A null Condition decodes as no Condition, so the Allow granted
unconditionally. A null Condition on a Deny only widens the veto and
stays accepted, as in the S3 Tables validator.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3tables: treat a NaN numeric condition value as unreadable
strconv.ParseFloat accepts "NaN", but no comparison with it holds, so a
Deny on NumericLessThan never applied and an Allow on NumericNotEquals
matched every request.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3tables: reject condition keys the evaluator never supplies
A condition on a key such as aws:SourceIp reads no request value, so a
Deny on it never applied and an Allow never granted. Accept only the
keys getConditionContextValues resolves, as the vector validator does.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3vectors: require Statement and Effect spelled as written
The evaluator decodes field names regardless of case, so an "effect" or
"statement" key slipped past the null Condition check and the other
raw-key checks while the stored Allow still granted.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3tables: treat an infinite numeric condition value as unreadable
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3tables: a stored Allow with a null Condition never grants
The typed decode reads "Condition": null as no Condition, so an Allow
stored before validation refused it still granted unconditionally, for
table and vector policies alike. Record the literal null while decoding
a Statement and skip such an Allow during evaluation.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3tables: validate a null Condition from the decoded Statement
validatePolicyDocument re-parsed the document to find a literal null
Condition; the Statement decode now records it, so the validator reads
the same flag the evaluator does and nullConditionStatements goes away.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3tables: keep Condition objects merged across key spellings
Decoding Condition through a json.RawMessage kept only the last of
"Condition" and "condition", so an empty object under one spelling
erased the other's restriction. Decode the statement as before and
probe for the literal null separately.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3tables: refuse a Condition split across key spellings
Condition objects under differently cased keys decode into one map, and an
operator they share keeps only the last spelling's keys, so a restriction
under the other spelling was dropped. Validation now refuses a statement
whose Condition is under more than one key, and one stored before that
never grants on an Allow and vetoes on a Deny.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3tables: refuse a Condition key repeated under the same spelling
PolicyDocument re-marshaled each statement through interface{}, and
Statement.UnmarshalJSON counted keys from a map, so both kept only the
last of a repeated Condition key. "Condition":null,"Condition":{} or two
Condition objects sharing an operator passed validation with a restriction
dropped. Statements now decode from their original bytes and the keys are
counted while walking the object.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3vectors: refuse a repeated Condition key in a vector bucket policy
The validator read Condition from a map, which keeps only the last of a
repeated key, so it accepted a statement the evaluator never grants on.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3tables: keep the re-marshaled statement decode, count Condition keys apart
Decoding statements from their original bytes let document order pick
between spellings of every field, where the re-marshal sorted the keys
first, so a stored "effect":"Deny","Effect":"Allow" became an Allow.
Statements decode as before again; only the Condition keys are counted
from the original bytes.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3tables: test that validation refuses a split Condition on a Deny
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3vectors: test that validation refuses a repeated Condition on a Deny
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* plugin worker: keep the stream's admin token out of job contexts
The bearer token minted when the worker opens its control stream was
attached to connCtx, which every detection and execution context derives
from. Job RPCs therefore forwarded that connect-time token, and when a
job added its own fresh token (s3_lifecycle's LifecycleDelete) the S3
server validated the stale one first and rejected the call once it
expired.
Attach the token only to the WorkerStream call, as the Rust worker does.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* credential: replace an inherited admin token instead of appending one
WithS3InternalAdminAuth appended its fresh Bearer token, so a context
that already carried an authorization value sent both, and the S3
gateway's checkAdminAuth validates only the first. Set the header on a
copy of the outgoing metadata so the fresh token is the only one sent.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* plugin worker: read the captured job metadata after the stream ends
Both result channels could be ready at once, letting select fail a
successful run. The handler reports before JobCompleted reaches admin,
so wait for the stream to finish and then read the captured metadata.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* plugin worker: test that dropping the connection cancels running jobs
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* s3: keep service account keys when a pushed identity replaces its parent
Service account changes push the parent user, whose proto carries only
the user's own keys. The merge replaced the cached parent with that
proto and dropped every service account key until a restart. Read the
parent's service accounts from the credential store so the push
rebuilds them, deleted and disabled ones included.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3: test that a parent identity push keeps its service account keys
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3: test a sibling service account survives a deletion push
Also check the restored secret and status, not just the key.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
seaweed-worker-core now depends on toml, but the seaweed-vector
workspace lockfile was not refreshed, so cargo build --locked fails
before the S3 Vectors integration tests start.
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* filer: clear a dead test replica's reachability record before closing it
The read path tries hosts that recently failed last. A dead test server
can land on a port an earlier test left marked as failed, so the live
replica answered first, nothing failed and the location refresh under
test never ran. Have the dead replica answer once through util_http so
its record is cleared before it closes.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* filer: close the dead test replica when reaching it fails
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* filer: clear the live test replica's reachability record too
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* master: tell a warming-up Assign client when warm-up ends
During warm-up the master refuses Assign with Unavailable and now attaches
the remaining warm-up as a gRPC RetryInfo detail. RetryWithBackoff may
shorten a backoff step to that hint plus 50 ms, never lengthen it, and only
for a valid, positive hint, so a writer retries right after warm-up ends
instead of up to one backoff step later. The warm-up itself is unchanged.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* master: jitter the retry-hint slack and hint LookupVolume too
Per review: the slack added to a server hint is now random in
[50ms, 300ms], so clients refused at the same moment do not all retry
in one instant. The warm-up refusal of LookupVolume carries the same
RetryInfo hint as Assign; the masterclient's lookup retry already goes
through RetryWithBackoff, so it lands at warm-up end too.
Tests: the slow guard tests are replaced by one sleep-free table test
of the step logic (zero, negative, huge, malformed, valid and wrapped
hints, a hint longer than the step, none). One timing test of the
shortened step stays, plus a jitter-bounds test and a LookupVolume hint
test. The no-hint control test, which was timing-sensitive, is dropped.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* master: name the shared warm-up test and file the lookup test by topic
Rename the no-remaining test to TestWarmingUpErrorWithoutRemainingHasNoHint,
since the helper now serves Assign and LookupVolume, and move the
LookupVolume hint test next to the other LookupVolume warm-up tests.
Tests only.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* util: read the retry hint through ServerStatus
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* master: bound the warm-up hint in tests by the remaining warm-up around the RPC
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* test: leave more scheduler margin in the retry-hint timing tests
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* util, master: trim retry-hint comments
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* test: widen the retry-hint timing bounds to just under the unhinted wait
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* test: require a refused warm-up call before the hinted assign retry
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* test: check the hinted retry wait through a test timer, not the wall clock
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* test: move the assign warm-up end to 2.6s for more hinted-retry headroom
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* test: require the hinted retry to wait out its timer
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
* admin: credit an unitemized slot to a pending job only after a later heartbeat
A pending reservation the worker does not list took one of the slots
the worker counts without itemizing, even when that heartbeat came
before the reservation and could not have counted the job. The slot
belonged to unknown work, so the admin undercounted it and could admit
one job too many, past a job type's per-worker limit, until the next
heartbeat.
A reservation now records when it was made, and claims an unitemized
slot only from a heartbeat received at or after that time.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* admin: time unitemized-slot credit by the heartbeat, not the last hello
A reconnect's hello moves LastSeenAt but keeps the old heartbeat, so a
pending reservation could claim a slot counted before it existed, and a
completed job's echo could end without a new heartbeat. Record when the
heartbeat itself arrived and compare reservations against that.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* admin: a completed job claims no slot from a heartbeat before its reservation
A job that completes before any heartbeat counts it has no echo, so the
slot the last heartbeat counted is other work. Crediting it to the job's
type let another job type pass its per-worker limit until the next
heartbeat.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* admin: count executor slots per job type, deduped against the heartbeat
per_worker_execution_concurrency is a per-job-type setting, but the
scheduler compared it against the worker's execution_slots_used, which
counts every job type. One s3_lifecycle job a worker reported running
after a reconnect made admin_script (per-worker 1) undispatchable on a
worker with three free slots. A reservation and the heartbeat entry for
the same job were also charged as two slots, so a per-worker limit of N
let N-1 jobs run.
Reservations now carry the job id and job type. The per-type limit is
checked against that type's work alone, the worker's slot total against
work of every type, and a heartbeat entry covered by a reservation is
counted once. Per-lane reservation pools go away: the per-type count is
what kept lanes from starving each other.
* admin: stop a lane waiting on executor capacity it cannot free
A queued job no executor could take was retried every backoff until
job_type_max_runtime ran out, up to 30 minutes, while the default lane
sat in executing and every other job type in it idled.
The dispatch loop now asks what holds the capacity. Slots held by the
lane's own jobs release when they complete, so it keeps waiting. Slots
held by other lanes or by work this admin never dispatched clear on a
schedule the lane has no view of, so the queued jobs are canceled at
once with a skipped_no_executor activity naming the worker, its slot
count and the job types holding it, and the next detection proposes
them again. A missing worker is waited for only the orphan grace.
* admin: cancel a job on its worker when the admin expires it
Expiring a job, by the stale sweep or by the operator's expire endpoint,
only failed the tracked record. The worker never heard about it and kept
running the job with its slot taken, so a job stuck in "starting" held a
slot for over a day and, with the old slot accounting, blocked a whole
lane.
Expiring a job now also sends the worker a cancel for it, so the slot
comes back on the next heartbeat.
* admin: count a completed job's heartbeat echo as its lane's own slot
The worker sends JobCompleted before the heartbeat that frees the slot, so
right after a dispatch goroutine releases its reservation the job still
shows in the worker's running work. That entry was counted as foreign, and
a lane with per-worker 1 gave up and canceled the rest of its queue after
the first job. A released reservation now stays for completedJobEchoGrace
and claims that heartbeat entry as the lane's own load.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* admin: let the admin's own jobs claim slots a worker does not itemize
The Rust worker reports only a slot count, every 10s, and the Go worker
counts a slot one heartbeat before it lists the job. Those slots were
charged on top of the reservations behind them, and once a job completed
its stale count read as foreign work, so the lane gave up its queue.
Unitemized slots now go first to this admin's unlisted reservations and
recent completions; the rest are of unknown type and count against every
type's per-worker limit, not only the slot total.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* admin: give up pinned jobs whose busy home a free worker cannot relieve
A job homed on a replica-host worker stays on that home while its queue is
shorter than LocalQueuePin, so a free remote worker cannot take it. The
outlook treated that free worker as a reason to wait, and a home held by
foreign work parked the lane for its whole window. A free worker now only
holds the wait for the orphan grace, after which the held homes decide.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* admin: let a redispatched job's reservation replace its released one
A job the worker turns away for capacity is released, kept as an echo, and
queued again under the same id. Once the new attempt ran, both the echo and
the new reservation matched its heartbeat entry and it counted as two
slots. Reserving a job now drops what an earlier reservation of it left.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* admin: credit a count-only slot to a completed job only until the next heartbeat
A released reservation claimed an unitemized slot for its full 30s grace,
so on a count-only heartbeat other work of another type was charged to the
completed job's type. A heartbeat that arrives after the release already
reflects the freed slot; from then on the slot is of unknown type.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* admin: name unitemized slots in the no-capacity reason
holders() listed only the job types the worker itemized, so a worker with
named and count-only work read as "2/2 slots used by <type>".
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* admin: keep the blocked clock running until a reservation succeeds
A popped job whose reservation then failed reset the clock before the
failure pushed it back, so pop-then-fail cycles never aged the block.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* admin: resend an expiry cancel the worker did not receive
Expiring a job sends its worker one cancel. When the worker is
disconnected, or its outgoing buffer stays full past the send timeout,
that cancel is dropped. The record is already failed, so later sweeps
and expire calls skip it, and a worker that kept the job running (a
detached job across a reconnect, or a jammed stream) holds the slot
until the job ends on its own.
The admin now holds an undelivered expiry cancel and sends it again on
the worker's next heartbeat that still lists the job. It is forgotten
when that heartbeat no longer lists the job, and skipped when the job is
active again, since a new attempt reuses the job id.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* admin: cancel an expired attempt by its request id
The expiry cancel named the job id, and a retry reuses it. The worker
keys its cancel funcs by both ids, so a cancel arriving after a retry
started on the same worker stopped the retry. The held resend also gave
up on any job that was active again, and the expired attempt's own late
progress marks the job running, so a worker that buffered progress
while disconnected kept the expired job.
The cancel now names the attempt's request id, as the other execution
cancels do, and the held resend no longer depends on the tracked state.
Only a cancel that can name nothing but the job id still skips a job
that is active again.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* admin: ignore late reports from an attempt the admin already ended
Expiring a job now cancels it on its worker, and the worker answers
with its own JobCompleted for that attempt. That overwrote the expiry's
failure with the cancel error and logged a second run record under the
same run id. Late progress from the expired attempt likewise reopened
the record as running.
A progress update or completion whose request id names an attempt the
record already closed is now dropped after the dispatcher's waiter is
told. A retry runs under a new request id, so its reports still apply.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* remote.uncache: skip a file overwritten since the walk read it
The uncache cleared the chunks of the entry its directory walk returned and
wrote it back unconditionally. A file overwritten after the walk read it
was reset to the remote pointer, and its new chunks were dropped, so an
acknowledged write was lost. Write back only while the filer still holds
the entry as read, and report the file as skipped otherwise.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* remote.meta.sync: keep files written locally after the directory listing
meta.sync lists each local directory once and works from that listing.
A changed remote file was written back over the listed entry
unconditionally, and a name missing from the listing was created without
O_EXCL, so a file written locally in between was replaced by the remote
pointer either way. Update only while the filer still holds the listed
entry, create exclusively, and report such files as skipped. The create
now also surfaces errors the filer returns in its response.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* remote.cache: keep files written locally after the lookup
The metadata step looks each file up and then either rewrites it from the
remote unconditionally or creates it without O_EXCL, so a local write
landing between the lookup and the write was replaced by the remote
pointer. Reuse doSaveRemoteEntry, which updates only while the filer still
holds the looked-up entry, create exclusively, and report such files as
skipped.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* remote.meta.sync: do not descend into a name a local file took
When the exclusive create of a remote directory finds the name taken, the
sync still descended into it, and creating the remote children under a
local file would turn that file into a directory. Descend only when the
name now holds a directory.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* remote.meta.sync: delete only the file version it listed
The type-change and -deleteStale paths removed a remote-backed file
unconditionally, so a local write landing after the directory listing was
deleted with it. Condition the delete on the listed mtime, as fs.verify
and volume.fsck do; the filer then keeps a file written since.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* remote.meta.sync: condition directory and mtime-less deletes too
The listed-mtime cutoff was only sent for files, so a remote-backed
directory replaced locally by a file between the emptiness check and the
delete was removed unconditionally, and so was a file listed without an
mtime. Send the cutoff for directories as well, and for an entry listed
without an mtime keep it once a write has set one.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* filer: test that DeleteEntry applies the request's mtime cutoff
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* remote.meta.sync: keep the listed nanoseconds as the cutoff of an epoch mtime
An entry listed at the epoch with a sub-second mtime was sent a 1 ns
cutoff, so the filer kept it unchanged although the old code deleted it.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* remote.meta.sync: report an entry the filer kept instead of deleting it
The filer answers a delete it skipped under the mtime cutoff with
success, so the sync printed (delete) for an entry a local write kept.
deleteLocalEntry now looks the name up after the delete and prints the
outcome, and a replaced file that was kept is not recreated.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* s3api: store and serve a bucket's PublicAccessBlock configuration
PutPublicAccessBlock, GetPublicAccessBlock and DeletePublicAccessBlock
answered NotImplemented. Store the configuration in the bucket entry's
extended attributes next to the ownership controls, carry it in the
cached BucketConfig, and answer Get on an unset bucket with
NoSuchPublicAccessBlockConfiguration like S3 does.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3api: refuse public ACLs when a bucket blocks them
With BlockPublicAcls set, a PutObject, CopyObject, CreateMultipartUpload,
POST Object, PutObjectAcl or PutBucketAcl whose canned ACL or grants name
the AllUsers or AuthenticatedUsers group is refused with AccessDenied.
Private and account grants keep working, so clients that send
x-amz-acl: private on every upload are unaffected.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3api: ignore public ACL grants for anonymous reads under IgnorePublicAcls
A bucket with IgnorePublicAcls no longer lets an AllUsers grant on the
bucket or on an object answer an anonymous read or listing. Bucket
policies and identity permissions are unaffected.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3api: reject PublicAccessBlock bodies with unknown elements
A misspelled element such as BlockPublicAcl was ignored, so the request
succeeded and stored the flag as false. Answer MalformedXML instead, and
store the configuration without the request's namespace.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3api: fail closed when a bucket's public access block cannot be read
A stored PublicAccessBlock that does not parse now blocks and ignores
public ACLs instead of acting as if none were set. PutBucketAcl and POST
Object return the bucket lookup error rather than skipping the
BlockPublicAcls check when the bucket config cannot be loaded.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3api: test BlockPublicAcls on copy, multipart initiate and a PUT-then-GET
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3api: keep stored public access block bytes on unrelated bucket updates
An unreadable stored configuration was replaced with the in-memory
all-true fallback whenever another bucket setting was updated, losing
the original value. Write the key only when the update changes the
public access block.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* volume: a read plan waits for a vacuum commit instead of answering 404
A vacuum commit closes the needle map, swaps the compacted files in and
reloads the map. It holds the data-file write lease for that and takes no
store lock. needle_read_plan, which the HTTP GET path uses, read the needle
map without the data-file read lease, so a GET planned in that window found
no map and answered 404 for a live needle.
needle_read_plan now takes the read lease first, as read_needle_with_option
does, so the plan waits for the commit and then reads the reloaded map.
Go's readNeedle does the same under dataFileAccessLock.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* volume: plan a read with the store guard released
A plan that waits on a vacuum commit's data-file lease under the store
read guard parks store writers (allocate, mount, EC ops) on that commit,
and every store reader queued behind them. Look the volume up under the
guard and plan after releasing it, as Go's readNeedle does, and look the
volume up again if its handle was closed while the plan waited.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* volume: retry a read plan while its handle keeps closing
A plan that waited out an unmount looked the volume up only once more,
so a replacement handle that also closed before its plan took the lease
answered NotFound for a needle a later mount holds. Repeat the lookup
while the planned handle is closed, at most three times, so a closed
handle that stays in the map still ends the loop.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* volume: start the swap-window check once the plan holds the volume
The swap-window test gave the plan 300 ms to answer early, so a reader
thread that started late passed the check without waiting on the
commit. Share the writer test's wait for the plan's volume handle and
start both timed checks after it.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* volume: run the swap-window read-plan test on a redb map too
A commit reloads the index through load_index_redb for the redb kinds,
a path the in-memory run never took during the swap window.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* volume: say a needle read plan is resolved under the data-file lease
The plan no longer resolves under the store guard, and resolving it
takes the data-file read lease.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* volume: read the needle through the plan in the read-plan tests
The tests checked only that the plan resolved and then read the needle
through Store::read_volume_needle, so a plan with a wrong offset or size
after the commit still passed. Read through the plan, as GET does, and
compare the data.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* volume: test the read plan's lookup after an unmount closed the handle
No test reached the closed-handle retry, so dropping it left every test
green. Take the lookup as a closure so a test can hand the plan the
handle an unmount closed and then the remounted volume, and check that
the plan reads the needle from the remount.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
* seaweed-common: share the security.toml search path with the volume server
* weed-worker: read jwt.filer_signing from security.toml and WEED_JWT_* like the Go worker
* helm: give the worker-lance container security.toml and the extra environment
* weed-worker: treat an empty signing key in the environment as unset
Viper ignores an empty environment value, so the Go worker falls back to
the security.toml key when WEED_JWT_FILER_SIGNING_KEY is blank. The Rust
worker took the blank value and connected with no token. An empty
--filer-signing-key / WEED_FILER_SIGNING_KEY also falls through to the
security.toml lookup now.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* weed-worker: read a quoted expires_after_seconds like viper
The Go worker reads jwt.filer_signing.expires_after_seconds with GetInt,
which also accepts a quoted number. The Rust worker dropped a quoted value
and minted tokens with no expiry.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* weed-worker: warn when security.toml cannot be read or parsed
Viper logs a warning when an optional config file is broken. The Rust
worker skipped it silently, so a typo looked like an install with no
signing key. Log the path and the parse message, without the quoted line,
which can hold a key.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* helm: keep POD_NAME and LANCE_NAMESPACE_URL out of worker-lance's extra environment
A duplicate name renders after the chart's entry and wins, silently
changing the Lance worker's --id or --namespace.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* weed-worker: list /usr/local/etc/seaweedfs in the security.toml search path
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* weed-worker: an explicit --filer-signing-key-expires-after-seconds wins
The flag was silently ignored whenever the key was discovered, from
WEED_JWT_FILER_SIGNING_KEY as well as from security.toml. Use it when
given, else the expiry configured beside the discovered key, else zero.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* weed-worker: test where filer_signing looks for security.toml
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* weed-worker: read a float expires_after_seconds as viper's GetInt does
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* weed-worker: cap the minted exp so a huge expiry cannot overflow
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
The fixture waited for the Lance catalog port and the engine, but weed s3
opens its HTTP port last, about half a second after the catalog here. The
open cluster's engine now registers within that window, so
test_engine_without_credentials met a connection refused on the go and rust
single legs. Wait for the S3 port like the other listeners.
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
The detection waiter selects on the request's completion channel and on
the stream's done channel. A worker that finishes a scan and then drops
its stream (a restart right after a scan is the usual shape) buffers the
completion into the request before the receive loop sees the EOF, and
the stream's cleanup then closes done. A waiter that reaches its select
after both are ready can take the disconnect branch, and a completed
detection, proposals included, is failed as "detection worker
disconnected" and retried or recorded as an error.
When the stream closes or the liveness tick finds the worker gone, the
waiter now takes a completion that is already queued and returns it as
the success it is; the detector lease still goes, since the worker that
answered is unreachable. A ctx that ended keeps its meaning: the caller
stopped wanting the answer. The wait loop moves into
awaitDetectionOutcome so a test can hand it both channels already
ready; before this change that test fails on about half its runs. A
completion that lands after the waiter gave up stays unmatched, since
the request's pending entry is gone and the next request has a new id.
* topology: refresh EC file and delete counts from heartbeats
UpdateEcShards only re-stored a heartbeated EC record when shard
membership changed, so FileCount and DeleteCount stayed at whatever the
first heartbeat after registration carried, while the volume server
reports fresh tombstone counts on every full EC heartbeat. Per-collection
object counts and EC vacuum candidate detection read the stored values.
Compare the counts per (volume, disk) against the record already in
hand and copy them onto the stored record in place, so the refresh
raises no new/deleted shard events and rebuilds no maps. A message
reporting 0/0 did not count (older volume servers, and the incremental
mount message omits the fields), so it keeps the stored values, also
across a membership-triggered rewrite. The incremental path's existing
branch picks counts up the same way.
* topology: swap EC shard records instead of mutating them in place
GetEcShards hands out the stored *EcVolumeInfo pointers and releases
ecShardsLock; CollectionStatistics, CollectionEcVolumeStats, ToDiskInfo
and the master's shard broadcast read fields through them afterwards.
The heartbeat count refresh wrote FileCount/DeleteCount into those
records under the lock, which the readers never take, so it was a data
race and a reader could pair a FileCount from one heartbeat with a
DeleteCount from the next. Disk.AddOrUpdateEcShard had the same write
on the incremental path, and its shard-bits/ratio merge was already
in place.
Stored records are now never written after they are published: a count
refresh, an incremental merge and a shard delete each build a copy
beside the stored record and swap the map entry under ecShardsLock, so
a reader holding the old pointer keeps one heartbeat's snapshot. The
refresh compares against whatever record is current under the lock,
since the incremental path may have replaced it since UpdateEcShards
looked. The 0/0-means-not-reported rule is unchanged.
The count tests now assert that the stored pointer changes while a
previously obtained one still reads its old values, and a -race test
runs full and incremental count refreshes against readers; it failed
on the previous code and passes now.
* weed-vector: look for security.toml in -config_dir first
weed processes search the directory -config_dir names before ., so a
cluster keeping security.toml there left the engine in plaintext.
--config-dir (also -config_dir, WEED_VECTOR_CONFIG_DIR) does the same.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* weed-vector: dial the masters the way weed clients do
The master's certificate must chain to grpc.ca and is not checked against
the host dialed, as weed clients verify (advancedtls.CertVerification), so
masters sharing one workload certificate, or dialed by IP, register as
they do from weed. Instead, grpc.client.master_allowed_server_commonNames,
grpc.client.allowed_server_commonNames or grpc.allowed_server_commonNames
(and the wildcard domain likewise) restrict which server identities are
accepted: the common name or a DNS SAN, or an exact URI SAN, as
security.Authenticator.AuthenticateServer does. These used to be refused
at startup; --tls-server-name is gone with the host check.
The TLS runs in a connector that reads the credentials asynchronously at
every dial, so the connect timeout still covers the reads, the TCP connect
and the handshake together. An IPv6 master's brackets are stripped before
the dial.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* test/s3vectors: the hardened leg allowlists server identities for grpc.client
The gateway and the engine then accept only servers named localhost, which
exercises the engine's allowed_server_commonNames check on the master dial.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* weed-vector: parse the refresh interval with time.ParseDuration's arithmetic
The interval was summed in floating-point seconds, so 0.9ns rounded up to
1ns where Go truncates it to 0 and falls back to the 5h default, 0.3s was
off by a nanosecond, and 3000000h was accepted where Go rejects anything
past the longest time.Duration. Each part is now an integer count of
nanoseconds plus the fraction Go adds, with Go's overflow bounds, and the
value is no longer trimmed, as Go does not trim it either.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* weed-vector: test a master dial whose credential read stalls
The bounded-dial tests passed no credentials or a prebuilt endpoint, so
moving the credential read out of the connect timeout would have failed
none of them. The new test points grpc.ca at a FIFO nobody writes.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* weed-vector: dial a closed port in the stalled credential read test
Against a silent listener, a dial that never read the credentials timed
out with the same message, so the test passed without the read. Nothing
listens on the port now, so only the stalled read can time out.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* weed-vector: test that Security::load searches the config dir
Only candidates() was tested, so load dropping config_dir on the way
would have passed.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* weed-vector: expand a leading ~ in -config_dir like weed's ResolvePath
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* weed-vector: hold the port in the stalled credential read test
A released port can be taken before the dial, and a dial that skipped the
read would then time out and pass. A listener that drops every connection
keeps the port and still fails such a dial at once.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
* admin: keep a reconnected plugin worker registered when its old stream unwinds
cleanupSession removed whatever session was registered under the worker
ID, so a worker that reconnected while its previous stream was still
unwinding lost the new session and its registry entry to the old
stream's deferred cleanup. Tie cleanup to the stream that is ending and
publish the new stream before the registry upsert so a lingering
predecessor sees itself superseded.
* admin: fail an in-flight detection when its worker disconnects
RunDetectionWithReport waited only on the caller's context and the
DetectionComplete channel, so a detection whose worker stream ended
mid-request could only finish by timeout. On a customer cluster a
planned restart of ec-vacuum-1 during an ec_vacuum detection left the
default lane in 'detecting' with every execution slot idle until the
3600s detection timeout, blocking every job type in that lane.
Bind the request to the stream it was sent on and fail the detection
as soon as that stream closes, whether by disconnect or by a reconnect
that replaced the session. A liveness tick also fails it once the
registry no longer considers the worker available (heartbeat stale),
sending a cancel in that case because the stream may still be open.
The detector lease is dropped so the next pick starts fresh, and a
'disconnected' detector activity is recorded. WorkerDisconnectForTest
now goes through the real session cleanup.
* admin: retry a scheduled detection once when its detector drops
When a detection fails because its worker disconnected, pick a detector
again and retry once within the same iteration while the job type's
runtime budget allows, instead of giving the whole cycle up. Any other
failure, or a failed retry, ends the run as before and the lane moves
on; the next cycle re-detects anyway.
* admin: register a plugin worker's hello and stream as one
step
WorkerStream published the stream under sessionsMu but ran the registry
upsert after releasing it, so two concurrent streams for one worker ID
could publish their hellos in the opposite order from their sessions:
the registry then described the older hello's capabilities while only
the newer stream was sendable, and detector picks routed on them.
registerSession upserts the hello and installs the stream under one
sessionsMu critical section, and a hello or heartbeat arriving on a
stream that is no longer the registered one is dropped instead of
rewriting the live stream's entry.
* admin: steer a detection retry off the stream that dropped
The retry picked while the lost worker could still be in the registry:
cleanupSession closed done before taking the registry entry down, and a
stream closed by a reconnect keeps its entry on purpose. The lost worker
ranks first on free slots, so the single retry went back to it and
failed on send while a healthy worker sat idle.
The disconnect error now carries the stream it came from and the retry
picks around it; pickDetector only returns workers whose current stream
is open, excluding registry entries whose stream is closed; and
cleanupSession removes the registry entry before closing done, so a
waiter woken by the closure never sees the dead worker as a candidate.
* weed-vector: read the cluster's security.toml
The engine finds security.toml where weed processes do (., ~/.seaweedfs,
/usr/local/etc/seaweedfs, /etc/seaweedfs) with WEED_* environment
variables over it, and builds gRPC client and server credentials by the
rules of security.LoadClientTLS and LoadServerTLS: a dedicated client
certificate is preferred, both halves of a pair must be set, and grpc.ca
is required once either is.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* weed-vector: gRPC TLS to the masters and from the gateway
The masters are dialed with [grpc.client]'s credentials when the cluster
has them, the master's certificate checked against the host dialed or
--tls-server-name. The engine's own listener takes its identity from
[grpc.vector] and then requires a client certificate chaining to grpc.ca,
which is how the gateway, dialing with [grpc.client], reaches it. The
rustls provider is pinned before the first TLS use, as the other binaries
linking both aws-lc-rs and ring do.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* scaffold: document [grpc.vector]
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* test/s3vectors: put every gRPC hop under mutual TLS in the hardened leg
VECTOR_TLS=1 now also writes [grpc.master], [grpc.volume], [grpc.filer],
[grpc.s3], [grpc.client] and [grpc.vector], and probes the master's and
the engine's gRPC ports: plaintext HTTP/2 and a client without a
certificate are refused, the client certificate gets a SETTINGS frame.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* test/s3vectors: give the test volume server 64 volume slots
Every collection grows 7 volumes and test_vector_bucket_encryption opens
two buckets while the previous test's bucket is still being torn down, so
16 slots were marginal: the Go legs failed CreateIndex now and then with
"failed to find writable volumes" and master.log showing "create 7
volume, created 4: Not enough data nodes found!".
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* weed-vector: reload the listener's certificates and enforce allowed_commonNames
The listener now runs the way weed servers run theirs. The certificate, key
and CA files are re-read every WEED_TLS_CERT_REFRESH_INTERVAL (weed's
default of 5h) and on SIGHUP, and new handshakes use the changed files
without a restart; a file that no longer parses is logged and the working
configuration stays. [grpc.vector] allowed_commonNames and
grpc.allowed_wildcard_domain restrict which client certificates are served,
checked against the subject common name as security.Authenticator and the
Rust volume server's CommonNameVerifier do.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* scaffold: allowed_commonNames and the refresh cadence for [grpc.vector]
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* test/s3vectors: the engine's allowlist keeps an outsider certificate out
The hardened leg sets [grpc.vector] allowed_commonNames = "localhost" and
probes the engine with a second certificate the same CA signed for
CN=outsider: the handshake must be refused, while the gateway's certificate
still gets an HTTP/2 SETTINGS frame.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* weed-vector: take the master's TLS name from the parsed endpoint
A master given without a port (https://master.example) had the whole URL
checked against its certificate.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* weed-vector: bound the master TLS handshake
connect_timeout covers only the TCP connect, so a master that accepted the
connection and never answered the handshake held up the other masters.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* weed-vector: look up security.toml keys the way viper does
An empty WEED_* variable no longer hides the file's value, and keys match
regardless of case, as for weed processes.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* weed-vector: refuse master allowlists it cannot enforce
weed checks a dialed server against allowed_server_commonNames and
allowed_server_wildcard_domain on top of grpc.ca. The engine's master dial
accepted every certificate the CA signed even when one was set; it now
fails at startup and names the key.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* test/s3vectors: read the whole frame header in the gRPC TLS probe
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* test/s3vectors: probe the volume, filer and S3 gRPC listeners too
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* weed-vector: read the master dial credentials only when given masters
A standalone engine never dials with grpc.client, so its settings no longer
stop it from starting.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* weed-vector: run each listener TLS handshake on its own task
The accept loop awaited every handshake before accepting the next
connection, so a peer that connected and sent nothing stopped every
gateway connection after it. Handshakes now run concurrently, each under a
10s timeout.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* weed-vector: read the master dial's TLS files at every dial
The client certificate, key and CA were read once at startup, so after the
files rotated a reconnect still presented the old pair. Each dial now reads
them, as the listener re-reads its own.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* weed-vector: read every unit Go durations accept in the refresh interval
WEED_TLS_CERT_REFRESH_INTERVAL values in ns, us or a leading + fell back
to five hours in the engine while weed servers honored them.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* weed-vector: fall back on a refresh interval too long to hold
An oversized WEED_TLS_CERT_REFRESH_INTERVAL panicked the engine at
startup; weed servers reject it and keep the default.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* weed-vector: refuse allowed_commonNames without the listener's certificate
With [grpc.vector] allowed_commonNames set but no cert and key, the
listener served plaintext and checked no client at all.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* weed-vector: read the master dial's credentials inside its connect timeout
The CA, certificate and key were read synchronously before the timeout
started, so a stalled credential path held up every other master.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* weed-vector: let a set key win over an empty spelling of it
With allowed_server_commonNames = "" beside allowed_server_commonnames
= "master", the lookup stopped at the empty exact match and the
allowlist went unnoticed.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
* admin: place uploaded files by their path inside an uploaded folder
The multipart file name keeps only the base name, so the upload API takes
an optional relativePaths field listing each file's path under the target
directory. Paths that are empty, absolute or contain . or .. components
are rejected per file.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* admin: upload a folder from the file browser
The upload modal gets a folder picker, and a folder dropped on the listing
is walked into, so its files upload with their subfolder paths.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* filer UI: upload a folder
An Upload Folder button picks a whole folder, and a folder dropped on the
listing is walked into. Each file is posted to its subfolder under the
current directory, which the filer creates on the first upload into it.
Progress rows are keyed by that path, so equal names in different
subfolders no longer share a row.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* admin: reject an empty relativePaths field instead of ignoring it
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* admin: upload a large selection in batches below the multipart part limit
Go's multipart parser rejects a form of more than 1000 parts, so a folder
of about a thousand files failed as a whole. The upload modal now sends
500 files per request and sums the results; progress covers all batches,
and the button stays disabled until the last one finishes.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* admin: escape file names in upload error messages
A failed upload reports the client-supplied file or folder path, and the
toast renders its message as HTML.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* admin: list failed uploads in the modal and reload the listing when it closes
A partial upload logged its failures only to the console before the page
reloaded, and a batch that failed after earlier ones succeeded left the
listing stale. The failures now stay in the open modal, and closing it
reloads the listing once any file was uploaded.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* admin: cancel the upload when its modal closes
Closing the modal, including with its Cancel button, left the batch loop
running, so the remaining batches were still written. The batch in flight
is now aborted, which also stops the loop, and the listing reloads since
the aborted request may already have stored files.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* admin: count upload progress by files when the selection has no bytes
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* admin: keep the newest upload selection when a folder drop finishes late
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* admin: cap the upload preview at a fixed number of rows
A folder of many thousands of files rendered one preview row per file.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* filer UI: upload a few files at a time
Every selected file started its own request at once, so a large folder
opened thousands of requests together.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* admin: regenerate the gzipped admin.js
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* admin: reload the listing after a batch fails without a result
A batch that fails mid-request may still have stored some of its files,
so closing the modal reloads the listing.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* admin: stop reading a folder drop once a newer selection replaces it
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* admin: skip the read error of a folder drop a newer selection replaced
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* filer UI: share the four-upload limit across selections
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* admin: regenerate the gzipped admin.js
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* admin: set up the file browser handlers once
admin.js already runs setupFileManagerEventHandlers on DOMContentLoaded,
so the file browser page registered every listener twice and each drop
was read twice.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* rust volume: run each gRPC TLS handshake on its own task
The incoming stream awaited every handshake before accepting the next
connection, so a peer that connected to a TLS gRPC port and sent nothing
stopped every gRPC connection after it. Handshakes now run concurrently,
each bounded by grpc-go's default 120s connection timeout, as a Go server's
are.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* rust volume: retry a failed gRPC accept instead of closing the listener
grpc-go backs off and retries temporary accept errors such as EMFILE.
The TLS incoming stream ended on the first accept error, which stopped
the gRPC port for the rest of the process lifetime.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* rust volume: test that the gRPC handshake timeout closes a silent peer
The handshake timeout becomes a parameter so a test can use a short one.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* rust volume: exit cleanly on SIGTERM and SIGINT
The shutdown path raced the HTTP and gRPC tasks with select!, then
awaited both handles again; tokio panics when a JoinHandle is polled after
it completed, so every signal ended in "JoinHandle polled after
completion" with exit status 101 and the volumes never closed. Waiting
for both tasks with join! is enough: both stop on the shutdown signal, and
a gRPC serve error sends it.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* test/volume_server: fail when a stopped server does not exit cleanly
The harness ignored the exit status after SIGINT, which is how the Rust
volume server's shutdown panic went unseen. Every server it stops must
now exit 0 within 10s. The Go volume servers get -preStopSeconds=0, which
the Rust ones already had, so a stop does not spend the default 10s grace
inside that window.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* test/volume_server: let assertArgPair find flags after a bare flag
rustVolumeArgs carries --volume.allowUntrustedRemoteEndpoints without a
value, so pairs no longer sit at even indexes and
TestRustVolumeArgsIncludeReadMode reported --readMode missing although it
was there.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* rust volume: stop both servers when either one exits
A gRPC task that panicked left the HTTP server and heartbeat running
until a signal arrived. Whichever server task ends first now sends the
shutdown and the other is awaited once, matching Go where a serve
failure is fatal.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* rust volume: report an HTTP task panic as a server error
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
* filer.sync: pass the replayed entry down to the chunk retry gates
The fetch, manifest-resolve, and manifest-upload retry gates only received the
replayed version's mtime, which is not enough to order it against the live
source entry. Pass the entry itself; the gates still compare mtime only, so
behavior is unchanged.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* filer.sync: stop chunk retries with the same supersession check the skip uses
The fetch, manifest-resolve, and manifest-upload retry gates stopped early only
for a gone or strictly newer source entry. A transient error therefore kept
retrying a version the source had rewritten under an older mtime until the
read recovered, while a live lookup that lags the replay could stop the
retries on its newer mtime alone. The error handler that runs once a gate stops
already decides with sourceMovedOn, which orders by commit timestamp and
compares chunk sets; the gates now use it too, and hasSourceNewerVersion has no
callers left.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* filer.sync: a lagging live lookup cannot prove a chunk dropped
A peer filer's event reaches subscribers before the source store applies it,
so the live lookup can return the version the replay replaces. That entry
never listed the replay's new chunk, so sourceDroppedChunk read a 404 on it as
an obsolete version and cut the missing-chunk wait short. Like sourceVersion,
it now ignores a live entry committed before the replayed version.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* filer.sync: test that a replicate failure on a superseded version is skipped
Covers onReplicateChunkError with a transient error for a deleted source entry,
a later commit with an older mtime, and an unchanged source, and points the
unconfirmed-supersession test comment at it instead of the helper test.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* filer.sync: test the manifest gate against a missing volume and a lagging lookup
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* filer.sync: a chunk the live source entry dropped is gone whatever its mtime says
sourceDroppedChunk refused to judge a 404'd chunk when the live source entry
carried an older mtime than the replayed version, treating that as a rollback.
Timestamp-preserving writers (cp -p, rsync -a, tar extracts) produce exactly
that order on every rewrite, so the vacuumed chunk sat in the retry loop for
the full missingSourceChunkGrace before being written off. Whether the live
entry still lists the chunk is independent of mtime order.
* filer.sync: treat a rewritten source entry as moved on regardless of its mtime
sourceVersion only compared chunk sets when the live entry was at least as
new as the replayed one, so a live entry stamped with an older mtime but
holding different chunks was never recognized as the current version. The
sink then fetched the replayed chunks, hit 404 on each vacuumed one, and the
same mtime guard in onReplicateChunkError kept the event from being skipped.
Whether the source rewrote the file is a chunk-set question, not an mtime
one; the live version is synced in its place as before.
* filer.sync: apply a substituted live snapshot even under a newer destination mtime
After sourceVersion swapped in the live entry, UpdateEntry re-ran
chooseUpdateAction on it and a destination whose mtime was above the live
entry's read as a late update, so the current source content was never
written. The live snapshot is the source's current state, not a replay, so
mtime order says nothing about it; repair from it unless the destination
already holds it.
* filer.sync: a live entry committed before the replay does not supersede it
Dropping the mtime guard let any chunk-set difference read as a rewrite,
including a live lookup that lags the event: a peer filer's event reaches
subscribers before the local store applies it, so the lookup can still
return the version the replay replaces. sourceVersion then synced that
older version over the destination and consumed the event. The meta-log
commit stamps order the two versions where mtime cannot, so a live entry
committed before the replay proves nothing.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* filer.sync: check commit order before the mtime supersession
A lagging live entry can carry a newer mtime than the replay, as when the
replayed version was stamped back to an older time, so sourceSupersedes
took it as newer and the older version was synced in place of the replay.
Rule out a live entry committed before the replay first.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* rust volume: count sorted-map files and deletes the way Go does
The Rust sorted needle map still used the counter rule Go dropped when
the index loaders were unified: every .idx row, tombstones included,
counted as a file, and each delete added one file and two deletions.
Go counts put rows as files and a delete as one deletion of the live
row's bytes, so the Go and Rust copies of one volume reported different
FileCount and DeleteCount for the same index.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* rust volume: test canDelete volume counters across restarts
Mirrors the Go store test: a canDelete volume reloaded from .sdx must
count its deletes, and the counters it reports must match what the next
reload computes.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* fix(storage): update deletion metrics in SortedFileNeedleMap.Delete (#12363)
* storage: test canDelete volume counters across restarts
A canDelete volume reloaded from .sdx must count its deletes so
volume.deleteEmpty and vacuum see it drain, and the counters it reports
must match what the next reload computes.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* shell: keep fs.mergeVolumes from reverting entries written during the merge
The walk reads a page of entries and moves their chunks one at a time, then
wrote each entry back unconditionally. A file overwritten in between got its
old version back, and the filer deleted the new version's chunks as garbage,
so an acknowledged write was lost without any error.
The update now carries IF_ENTRY_EQUAL over the entry as the walk read it. When
the filer refuses because the entry changed or was deleted, the copies made on
the target volumes are deleted, their plan room is released, and the entry is
skipped; the source needles stay where the filer still points.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* shell: keep fs.distributeChunks from reverting a file overwritten while it runs
The command looks the entry up once, moves its chunks, and then wrote that
entry back unconditionally, so an overwrite during the redistribution was
reverted and the filer deleted the new version's chunks as garbage.
The update now carries IF_ENTRY_EQUAL over the looked-up entry. When the filer
refuses because the file changed or was deleted, the moved copies and any new
manifest chunks are deleted and the command reports it.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* shell: keep s3.versions.audit -heal from clearing a pointer that just moved
The heal cleared the latest-version pointer on the directory entry the walk
read, unconditionally. A version written between the audit and the heal has
already pointed the directory at a live file, and clearing that pointer hid it.
The heal now carries IF_ENTRY_EQUAL over the audited entry and reports a
refusal as a failed heal.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* shell: keep fs.meta.changeVolumeId from reverting entries written meanwhile
The walk rewrote each entry's volume ids and wrote back the copy it had read,
unconditionally, so an overwrite between the read and the write was reverted
and its chunks were deleted as garbage. The update now carries IF_ENTRY_EQUAL
over the entry as read, and a refused entry is skipped and reported.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* shell: keep the target copies another fs.mergeVolumes run committed
moveChunk keeps the key and cookie, so two overlapping merges of the same
source into the same target write the same fid. When one run commits, the
other's update is refused, and deleting its copies deleted the needle the
committed entry now points at.
A refused update now re-reads the entry, resolving its manifests, and deletes
only the copies it no longer references. If the entry cannot be read, the
copies are left for volume.fsck.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* shell: let a later hard-link sibling retry a merge the filer refused
The first sibling of a HardLinkId claims the move for all of them. When its
update is refused, the claim made every later sibling skip, so the shared
chunks stayed on the source volume for the rest of the run. A refused update
now releases the claim, and a sibling listed after the change retries the move
against the entry as it is now.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* shell: delete the manifest blobs a failed fold in fs.distributeChunks saved
When re-manifestizing failed partway, the blobs already uploaded were only
counted in a log line and stayed on their volumes with nothing referencing
them. They have freshly assigned fids, so they are now deleted with the
cookie-checked delete the other orphan cleanups use.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* shell: check the heal clears the pointer file name too
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* shell: require the shell lock for fs.mergeVolumes -apply
moveChunk keeps the key and cookie, so two overlapping runs that move the same
chunk into the same volume write the same fid. Re-reading the entry before
deleting a refused run's copies covers the run that already committed, but not
one that commits after that read. With the lock, two applying runs can no
longer overlap, as volume.fsck and volume.merge already require. A dry run
still works without it.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* shell: check unreferencedTargets reads the refused entry itself
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* shell: check unreferencedTargets keeps a copy a manifest references
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* shell: check unreferencedTargets keeps the manifest copy and skips an unreadable manifest
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* fix(fsck): inspect references to deleted volumes
* volume.fsck: treat every match-all collection filter as global
ALL_COLLECTIONS and EACH_COLLECTION compile to the same match-all
matcher as an empty filter or "*", but the string check treated them
as a restricted scope: a selected missing volume was rejected and
references to master-absent volumes were reported without being purged.
Use the compiled matcher instead.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* volume.fsck: purge an entry whose manifest is on a missing volume
A top-level chunk on a missing volume already proves the entry is
unreadable, whether or not its manifests resolve. Skipping such entries
kept the very case where the manifest needle sat on the retired volume,
and the remaining unresolved manifest then blocked every later orphan
purge. The missing-needle check on live volumes never had this gate, so
both paths now treat these entries the same. The warning now says only
orphan purging is disabled, which is the only thing the counter gates.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* volume.fsck: report a failed Filer delete
A rejected purge DELETE printed nothing unless -v was set, so the run
looked successful after "deleting path" while the entry stayed. The
filer answers 204 for an entry that is already gone, so only real
failures print.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* volume.fsck: tighten help text, comments and missing-volume tests
The help now wraps like the rest of it and says that a purge removes
the whole entry, including its chunks on other volumes. The live test
header no longer calls the run report-only, and the reverse-scan test
checks the report line for each missing chunk.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* volume.fsck: drop the missing-volume design notes
The notes said no destructive change was approved while the change adds
a confirmed purge, still described the dropped manifest gate, and
mostly recorded local runs and a reverse index that is not part of
this change. The operator contract lives in help volume.fsck.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* volume.fsck: allow a report-only missing-volume scan with -collection
The collection check exists to stop a purge that cannot be scoped, but
it also rejected a report-only run that deletes nothing. Apply it only
with -reallyDeleteFilerEntries, and limit the -confirmMissingVolumeDeletion
description to the -volumeId case it guards.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* volume.fsck: count only a 2xx answer as a purged entry
A redirect the client could not follow left the entry in place but was
recorded as deleted, which also queued its parent for empty-directory
cleanup.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* volume.fsck: say that a filtered purge of a missing -volumeId is refused
The help and the early warning both said a purge with -collection only
reports references to unlisted volumes, but a -volumeId naming one is
rejected before the scan. The help now says so, and the warning prints
after that check, so a refused run no longer claims it will report.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: MZ01-CHOIMINSU <mz01-juwannam@MZ01-CHOIMINSU.local>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* admin: scope plugin job and activity fetches to the selected job type
The plugin page fetched the newest 500 jobs and 500 activities
cluster-wide and filtered them by job type in the browser. On a busy
cluster that window can hold none of the selected type (433
erasure_coding + 64 ec_balance jobs, 497 ec_balance activities), so
/plugin/lanes/default/execution?job=ec_vacuum showed "No execution
jobs tracked yet" while the server had seven recent ec_vacuum jobs.
When a job-type tab is active, fetch the queue/detection/execution
lists with job_type=<selected> so the server filters before applying
the limit, and refetch when the selected job type changes. The
unscoped lists are still fetched for the status cards and the overview
per-type summary.
* admin: drop a plugin monitor refresh whose job type is no longer selected
A refresh started on one job-type tab could finish after the user moved
to another tab and that tab's refresh had already landed. Its rows, all
of the earlier job type, then replaced the current tab's and the
renderers filtered them down to empty tables until the next refresh.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* admin: show the lane-wide rows when a job-type scoped fetch fails
On a job-type tab a failed scoped jobs or activities request threw before
anything was stored, so the status cards, scheduler state and tables all
stayed stale. Fall back to the unscoped rows, which the renderers already
filter by job type.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* admin: filter plugin jobs and activities by lane before the limit
/api/plugin/jobs and /api/plugin/activities took the newest N rows across
every lane and then dropped the other lanes' rows, so on a busy default
lane a quieter lane's page could come back empty.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* admin: keep a failed plugin monitor fetch's last rows and report it
Falling back to the lane-wide rows when a scoped fetch failed could still
blank the tables, since that window may hold none of the selected type,
and the failure went unreported. A lane-wide or scheduler-state failure
also threw before the scoped rows that did arrive were stored.
Each request now keeps its previous value on failure: the scoped rows
only while the tab is unchanged, otherwise the lane-wide rows. Whatever
arrived is stored and rendered, then the first failure is rethrown so the
caller reports it.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* admin: report a monitor refresh failure apart from the action it follows
expireJob and runWorkflow refreshed the monitor inside the action's try
block, so a failed refresh after a successful expire or workflow run was
reported as a failed action, and the expired job's detail was not
reloaded. Catch the refresh separately, as runDetection already does.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* admin: show a repeated plugin page refresh failure once
refreshAll runs every 3 seconds and notify opens a modal, so a request
that kept failing reopened the same alert on every tick. Report an error
only when it differs from the last one, and forget it once a refresh
succeeds.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* admin: summarize the EC delete_count accounting warning per refresh
collectCollectionStats runs on every admin collection, bucket and
topology refresh, and it logged a Warning for each EC volume whose
delete_count summed across shard holders exceeds its file_count. That
sum is an expected accounting artefact (a tombstone can be recorded on
more than one holder), it recurs identically on every pass, and on one
cluster with hundreds of such volumes it grew the admin log to 20 GB,
more than 99% of it this one line.
Log one V(1) summary per pass with the affected volume count and keep
the per-volume detail at V(2). The object count is still skipped for
those volumes.
* admin: test that an EC volume with deletes above file_count is skipped
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* test/s3vectors: a hardened cluster, with TLS volume servers, read JWTs and encrypted chunks
Three switches on the test cluster: VECTOR_TLS=1 puts the volume servers
behind HTTPS requiring client certificates and enables https.client for
every process; VECTOR_READ_JWT=1 gives every process a jwt.signing.read.key,
so volume reads need a token; VECTOR_ENCRYPT_VOLUMES=1 starts the gateway
with -encryptVolumeData. The integration matrix gains a hardened leg per
volume server with all three on.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* test/s3vectors: the hardened cluster proves its enforcement
Once the gateway is up, the harness checks that the hardening holds, not
only that it is configured: a TLS volume server refuses plain HTTP (Go
with HTTP 400, Rust with a TLS alert) and a connection without the client
certificate, and answers with it; under a read key an unsigned read of a
real chunk, written through the gateway and located through the filer,
gets 401.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* test/s3vectors: the hardened cluster proves its chunks are encrypted
With VECTOR_ENCRYPT_VOLUMES the probe reads the stored chunk straight
from the volume server, signing the read when a read key is set, and
requires a recorded cipher key and bytes that differ from the plaintext.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* test/s3vectors: the signed probe read must return the chunk
An empty 200 passed both branches; an unencrypted read now has to return
the bytes written, and an encrypted one has to return something else.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* test/s3vectors: a read key in the caller's environment does not leak into the cluster
Only VECTOR_READ_JWT turns read JWTs on; an inherited
WEED_JWT_SIGNING_READ_KEY made the volume servers refuse the probe's
unsigned read in an encrypt-only run.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* test/s3vectors: quote certificate paths as TOML strings in security.toml
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* test/s3vectors: keep non-ASCII certificate paths as UTF-8 in security.toml
TOML has no surrogate-pair escapes, and the Rust volume server reads the
quoted value without decoding escapes at all.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* test/s3vectors: an encrypted probe chunk must not be a readable gzip stream
The gateway may gzip the body before encrypting it, so a chunk uploaded
compressed but unencrypted would differ from the body and still pass.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
* s3api: a known-answer fixture generator for SSE-S3 and filer-encrypted chunks
TestGenerateDirectReadFixture writes, from this package's own encryption,
the pieces a reader outside the gateway must handle: SSE-S3 chunks with
their per-chunk metadata, the object-level key, a single-stream ciphertext,
the KEK derived from s3.sse.key and wrapped under a passphrase in both
on-disk formats, and chunks encrypted by the filer, plain and gzipped. It
is skipped unless DIRECT_READ_FIXTURE names the output file.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* s3api: the fixture generator restores the key manager and checks the write
The test set the global key manager's KEK and left it; a later test in
the same run then found a signing key where it expects none. The previous
key is restored with t.Cleanup. The fixture file's close error is checked
too, so a delayed write failure fails the test instead of leaving a short
fixture.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* s3api: the fixture generator unwraps both KEK formats before writing
The legacy v1 wrap is built by hand in the test; reading v1 and v2 back
through unwrapKEK ties both to the format the key manager accepts.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3api: pin a legacy v1 wrapped KEK made outside the package
Generating the v1 sample through legacyKEKWrappingSalt let a change to
the salt pass on both sides; a vector wrapped elsewhere under the
historical salt must still unwrap.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
* fix(s3api): reject an invalid storage class before uploading the body
* fix(s3api): reject an invalid storage class on CreateMultipartUpload
* fix(s3api): validate storage class and tags before reading the PUT body
* fix(s3api): validate storage class and tags with the other PUT headers
Checking them next to ValidateRequestEncryption keeps the check ahead of
any later hook that may answer the PUT without reaching putToFiler.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* refactor(s3api): reuse validatePutHeaders in putToFiler
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* fix(s3api): store storage class and tags on directory-marker PUTs
The marker path validated x-amz-storage-class and x-amz-tagging but its
mkdir callback never wrote them, so a valid value was accepted and then
dropped and GetObjectTagging on the marker came back empty. Store both
the way putToFiler does.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* test(s3api): read directory-marker tags back through GetObjectTagging
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* filer: say jwt.filer_signing.key gates only the IAM gRPC service
The key never covered the rest of the filer gRPC port, which can read and
write the IAM store under /etc/iam. Point operators at [grpc.filer] mTLS for a
port reachable beyond a trusted network instead of the key.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* filer: note the IAM gRPC OIDC provider and role RPCs are refused without the key
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
At BestSpeed, Go 1.26's flate writes 4000 random bytes plus a 300-byte
zero run as a stored block, so the gzip output was 28 bytes larger than
the input and the test failed its own precondition on the toolchain
go.mod pins. With 450 zero bytes the saving is 5.7% on Go 1.26 and 8.4%
on Go 1.27, inside the marginal range on both.
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* s3tables: resolve s3vectors:VectorBucketTag like s3tables:TableBucketTag
Both name a tag on the bucket that holds the target, whichever resource
the request acts on.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* s3vectors: give vector policy conditions the bucket, index and request tags
A vector bucket policy is evaluated with these condition keys:
- aws:ResourceTag/<key>: the tags of the bucket, or of the index when the
request names one.
- s3vectors:VectorBucketTag/<key>: the bucket's tags, for bucket and
index requests alike.
- aws:RequestTag/<key> and aws:TagKeys: the tags TagResource and
CreateIndex set, and the keys UntagResource removes.
An index's tags are read only when the policy has an aws:ResourceTag
condition. The bucket's come from the lookup that already reads its
policy.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* s3vectors: accept tag conditions in vector bucket policies
PutVectorBucketPolicy rejected every Condition, since no condition key
was supplied. A condition is now accepted when its operator is one the
evaluator knows and each key is aws:ResourceTag/<key>,
aws:RequestTag/<key>, aws:TagKeys or s3vectors:VectorBucketTag/<key>.
Any other key would never match, so it is still rejected, as is a
condition field not spelled Condition.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* test/s3vectors: cover tag conditions in vector bucket policies
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* s3vectors: reject condition values the evaluator cannot compare
A null condition value was accepted and then panicked the request that
matched its statement, a Deny reaching even the bucket owner. A value
must now be a string, number or boolean, or a non-empty list of them.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* admin: carry a file browser upload size limit
The admin server keeps the total size limit of one file browser upload, 500 MB by default as the UI enforced before, and hands it to the file browser page.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* admin: reject file browser uploads over the size limit
The 500 MB limit was only checked in the browser, so any client could post larger uploads and fill the temp dir with spooled parts. Cap the request body at the limit plus room for the multipart framing, and answer 413 when the files add up to more than the limit.
Parse the form before reading the path field. FormValue used to parse it first with the default 32 MB memory, which made the later 1 GB ParseMultipartForm a no-op; keep 32 MB explicitly.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* admin: show and check the configured upload limit in the file browser
The upload dialog reads the limit from the page instead of a hard-coded 500 MB, lists it under the file picker, and skips the check when uploads are unlimited.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* admin: add -uploadMaxSizeMB to set the file browser upload limit
weed admin -uploadMaxSizeMB and weed mini -admin.uploadMaxSizeMB set the total size limit of one file browser upload; 0 removes it.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* admin: note why the upload framing allowance is enough
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* admin: test an upload whose files only exceed the limit together
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* admin: remove spooled upload parts when the upload handler returns
Parts over 32 MB spool to temp files. The router passes the handler a copy of the request, so net/http never sees the parsed form and never removes those files. Every large upload, and every one rejected for size after parsing, left them on disk.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* s3vectors: run an operation for trusted local tooling
Like s3tables.Manager.Execute, Server.Execute runs one S3 Vectors operation
through its handler as a given identity, without a signed request. The admin
console can then manage vector buckets through the same validation,
authorization and catalog steps as the API.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* admin: list, inspect, create and delete vector buckets over the API
/api/s3vectors runs the S3 Vectors operations through Server.Execute as the
admin account: list vector buckets with their encryption, read one bucket
with its policy and each index's data type, dimension, distance metric and
non-filterable keys, create and delete a bucket, and delete an index. Going
through the operations keeps the API's rules, such as ICEBERG buckets not
being vector buckets and the implicit index namespace not blocking a delete.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* s3vectors: export the rule for which table buckets are vector buckets
IsVectorBucketFormat names the check ListVectorBuckets and isVectorBucket
already make, so the admin console can tell the same buckets apart.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* admin: vector bucket pages
Object Store > Vector Buckets lists the vector buckets with their ARN,
encryption and creation time, creates and deletes them, and opens one bucket
with its indexes (data type, dimension, distance metric, non-filterable keys)
and its policy, where an index can be deleted. A LANCE row on the Table
Buckets page links to the same bucket's vector page, since both APIs serve it.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* admin: stop the vector bucket and index lists at the list limit
A page cut short by filtered ICEBERG buckets let the next page carry
the list past s3TablesAdminListLimit.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* admin: say when a vector bucket or index list is cut at the limit
The lists stop at s3TablesAdminListLimit while more remain, so mark
them truncated and point to an S3 Vectors client for the rest.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* admin: indent the vector bucket policy
A policy stored as one line ran off the right edge of the details page.
Policies that are not valid JSON are shown as stored.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* admin: show a deeply nested vector bucket policy as stored
Indenting multiplies the size of a deeply nested document, so a policy
past 32 levels of nesting is shown without indenting.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* volume(rust): replicate the stored needle instead of the request body
The primary forwarded the original request body and headers, so a
replica re-derived compression from the raw bytes. Send what Go's
ReplicatedWrite sends: a one-part multipart upload of the stored bytes,
gzip-marked when compressed, with the name, mime, pairs and Content-MD5,
and carry ts, ttl, cm and fsync in the query so form-field values reach
the replica too.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* volume(rust): keep the primary's compression on replica writes
A type=replicate upload now stores the bytes as received, like Go's
ParseUpload, so a replica no longer compresses data the primary stored
uncompressed.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* volume(rust): replicas keep the primary's name and mime
Forward the primary's filename and mime even past the 255-byte needle
fields, so a replica applies the same limit and sets the same flags,
and skip the filename MIME guess on type=replicate, so an untyped
upload stays untyped on every copy.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* volume(rust): keep the multipart part Content-Type as sent
Parsing the part type as a mime value lowercased it and dropped values
it could not parse, so a replica stored a different mime than the
primary. Read the header verbatim, as Go's part.Header.Get does.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* volume: replicas keep the primary's mime as forwarded
A Go replica guessed the mime from the file extension, so an upload the
primary stored as application/octet-stream, or with no mime at all, got
a different type on the replica. On type=replicate keep the part's
Content-Type as sent, and have the replication sender leave an empty
mime empty instead of filling it in from the extension. This matches
what the Rust replica does.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* volume(rust): pick the JPEG orientation fix by file name, as Go does
Go checks the mime and the upload's file name, Rust checked the mime and
the URL path, so the same upload could be rotated on one server and not
on the other. With the name and mime forwarded as stored, a replica now
reaches the primary's decision, and a rotated image has no EXIF tag left
to act on, so the fix stays a no-op there.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* s3vectors: tag vector buckets and indexes
TagResource, UntagResource and ListTagsForResource answer on
/tags/{resourceArn} (POST, DELETE, GET) and by name. Each authorizes its
own s3vectors action on the bucket or index ARN, then reads or writes the
catalog record's tags. A resource holds at most 50 tags.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3vectors: apply the tags passed to CreateVectorBucket and CreateIndex
The tags field was accepted and dropped. It is now stored on the new
bucket or index, and creating with tags needs s3vectors:TagResource in
addition to the create action.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* test/s3vectors: cover resource tagging end to end
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3tables: keep tag operations to a bound table bucket
TagResource, UntagResource and ListTagsForResource run through Execute
with WithTableBucketCreatedAt now answer not found when the bucket name
holds another bucket, as the bound creates and deletes already do.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3vectors: bind tagging to the authorized bucket and to vector indexes
The catalog tag calls carry the creation time of the bucket authorizeVector
approved, so a bucket created again under the name is not touched. An index
ARN must name a Lance table: another table in the vector namespace of a
format-less bucket is refused, as DeleteIndex does.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3vectors: match a TagResource grant scoped to the bucket being created
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3tables: merge tag writes into the tags the record holds at the write
TagResource and UntagResource merged into the tags they read before the
compare-and-swap and wrote that map on every attempt, so a tag write that
landed in between was undone. The merge now runs inside the mutation.
WithMaxTags lets an in-process caller cap the merged set there.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3vectors: check the 50-tag limit inside the catalog's tag write
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3tables: check a tag operation's bound bucket after its reads
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3vectors: validate each tag key UntagResource removes
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3vectors: validate untag keys one at a time
UntagResource sized a map by the number of submitted tag keys before
validating them, so a large body of repeated keys allocated far more than
the request itself, before authorization ran.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3vectors: authorize UntagResource before validating its keys
Validating a near-32 MiB body of tag keys costs about 13x its JSON
decode, so an unauthorized caller could make the gateway spend that
work on a request it then refuses.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* filer: test the rename subdirectory check through CanRename
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* filer: reuse IsEqualOrUnder for the rename subdirectory check
A trailing slash on the old name made the source path end in a slash,
so a child of the source was no longer seen as under it.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* compression: compress JSON uploads, and match file types by extension
IsCompressableFileType did not recognise JSON media types, so S3 uploads of application/json (which carry no file name) were never compressed. Its callers also passed filepath.Base of the file name where it expects the extension, so the .json/.js/.html/... extension rules never matched a real file name. Both callers now pass filepath.Ext; the function lower-cases the extension and media type, ignores media-type parameters ("application/json; charset=utf-8"), and treats */json and *+json types as compressible.
* compression: treat application/json-seq as JSON
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* volume(rust): compress JSON uploads and match media types without parameters
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* operation: keep uploader gzip only when it saves at least 10%
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Pat White <5015863+patwhite@users.noreply.github.com>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
CanRename rejected a move whenever the new parent directory's path started
with the source path as a plain string, so moving /dir/a into /dir/a.d
(or anything under /dir/ab) failed with "can not move directory to a
subdirectory of itself". Check the path boundary instead: reject only
when the new parent is the source itself or lies under it.
Claude-Session: https://claude.ai/code/session_01VncHKTehVroSAzRMUHykEX
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* s3tables: bind DeleteNamespace to the authorized namespace
The delete claimed whatever finished namespace held the name at claim
time, so one deleted and created again by another owner after the
authorization read was removed under the first one's authorization.
The claim now requires the creation time that was authorized.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3vectors: bind DeleteVectorBucket's namespace cleanup to the authorized bucket
After the bound DeleteTableBucket reported the bucket not empty, the
index namespace cleanup ran unbound, so a bucket deleted and created
again in between lost its empty index namespace under the first
bucket's authorization. DeleteNamespace now honors the bucket binding,
checking the bucket after reading the namespace, and the cleanup
carries it.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
A merged or closed PR kept every pull_request run going: concurrency groups
cancel a run only when a newer push arrives. After #12329 merged, 16 of its
runs were still queued or running. On pull_request_target closed, cancel the
queued and running pull_request runs on the PR's head branch whose commit
belongs to the PR.
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* s3tables: bind table bucket policy writes to the authorized bucket
PutTableBucketPolicy compared only the policy bytes, and
DeleteTableBucketPolicy removed the policy unconditionally. A bucket deleted
and created again under the same name between authorization and the write
took a policy change nobody authorized for it. Both writes now require the
bucket's CreatedAt and the policy they were authorized against, answer 409
otherwise and 404 for a bucket that is gone.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3tables: bind bucket tag writes to the authorized bucket
TagResource and UntagResource checked the creation time only for a table
ARN. On a bucket ARN the merged tags landed on a bucket deleted and created
again since authorization. extractResourceOwnerAndBucket now returns the
resource's CreatedAt for both kinds, and the write requires it.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3tables: bind bucket maintenance writes to the authorized bucket
PutTableBucketMaintenanceConfiguration wrote onto whatever bucket held the
name at write time. authorizeMaintenanceBucket now returns the bucket's
CreatedAt like authorizeMaintenanceTable, and putMaintenanceConfiguration
takes the check and creation time from both callers instead of an optional
closure.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3vectors: bind vector bucket policy writes to the authorized bucket
Put/DeleteVectorBucketPolicy authorized inside the compare-and-swap with
the owner from an earlier bucket read, so a bucket deleted and created
again by another account in between had its policy rewritten by the old
owner. The mutation now requires the entry's createdAt to match the
record it authorized against.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3vectors: bind DeleteVectorBucket to the authorized bucket
DeleteVectorBucket authorized one bucket record, then ran the trusted
catalog DeleteTableBucket on whatever bucket held the name by then, so an
empty bucket deleted and created again by another account in between was
deleted by the old owner. The catalog delete now accepts the authorized
bucket's createdAt through an in-process context value and treats any
other bucket as gone.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3vectors: bind CreateIndex and DeleteIndex to the authorized bucket
Both authorized the bucket, then ran trusted CreateNamespace, CreateTable
or DeleteTable on whatever bucket held the name by then, so an index could
be created in, or deleted from, a bucket another account created again in
between. Those catalog operations now honor the bucket binding
DeleteTableBucket already takes, and the vector handlers pass the
authorized bucket's createdAt.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* s3tables: bind table and view writes to the record they were authorized against
* s3tables: bind UpdateNamespace to the namespace it was authorized against
UpdateNamespace writes back the whole record it read, owner and creation
time included. A namespace deleted and created again before the write took
the first one's owner and CreatedAt, which also undoes the CreatedAt binding
creates now rely on. Refuse the write with 409 when the namespace at the
path has another CreatedAt.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
PUT and DELETE ?lifecycle were authorized as Write, so an identity granted
only object writes on a bucket could add expiration rules or remove the ones
the bucket administrator set. They now need Admin on the bucket, like the
encryption, public access block and ownership controls writes. Identities
authorized by IAM policies are unaffected: they are still checked against
s3:PutLifecycleConfiguration.
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
SigV4 verification picked the 32 MiB S3 Vectors body limit from the
credential scope, which is not verified until the body is hashed. A request
claiming the s3vectors scope on any other route could make the gateway buffer
32 MiB before authentication instead of 10 MiB. The vectors handler now sets
the limit on the request context, and every other route keeps 10 MiB.
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* s3api: hash an s3vectors body up to the Vectors API limit
SigV4 verification of a non-S3 request hashed at most 10 MiB of the body
and silently dropped the rest, so any S3 Vectors request over 10 MiB failed
its signature check with 403, although the API accepts 32 MiB. The limit
now follows the signing service, and a body over it is reported as
ErrEntityTooLarge instead of being truncated; the embedded IAM answers that
with InvalidInput, as it already does for a body over its own limit.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* s3vectors: answer an oversized request with a ValidationException
A body over 32 MiB was refused as "403 AccessDenied: authentication
failed". A declared Content-Length over the limit is now refused before
authentication, and a streamed body over it, whether the signature check
or the handler reads it, gets the same "request body exceeds 32 MiB"
ValidationException. The limit is the one SigV4 verification uses.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* s3vectors: log engine and catalog failures at the default level
An engine error reached the client as a bare "internal error", and the
cause was logged only with -v=1, so a misconfigured engine left nothing in
a default log. Engine and catalog failures are now logged as errors, and a
reported unavailability as a warning, with the operation, bucket and
index. The response still leaves the engine's message out, since it can
name internal hosts and paths, and points at the gateway log instead.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* s3api: hash the signed body through readRequestBody
streamHashRequestBody repeated readRequestBody's read-past-the-limit
check; it now reads the body with readRequestBody and hashes the bytes.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* s3api: encrypt the objects of a vector bucket declared AES256
CreateVectorBucket recorded encryptionConfiguration sseType AES256 and
returned it, but nothing on the write path read it: the index's data
files, manifests and transactions were stored without SSE.
A LANCE table bucket's record now supplies the bucket's default
encryption whenever the bucket has none of its own, so PutObject,
CreateMultipartUpload and CopyObject apply SSE-S3 to every object the
engine writes. The S3 bucket metadata itself is unchanged.
ICEBERG table buckets keep recording the declaration only: their
catalog reads the manifests clients upload straight from the filer,
which would see ciphertext.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* s3tables: refuse an AES256 vector bucket on a gateway without an SSE-S3 key
A vector bucket's AES256 is now applied to every index write, and SSE-S3
fails each write when the gateway has no KEK. Creating the bucket there
succeeded and the first CreateIndex came back as InternalFailure.
CreateTableBucket now refuses a LANCE bucket declaring AES256 when the
gateway reports no SSE-S3 key, naming s3.sse.kek and s3.sse.key;
CreateVectorBucket returns it as ValidationException. ICEBERG buckets,
whose declaration is only recorded, and tooling that sets no checker
are unaffected.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* s3api: refuse DeleteBucketEncryption on a bucket's declared encryption
GetBucketEncryption reports a vector bucket's declared AES256, but
DeleteBucketEncryption answered NoSuchBucketEncryptionConfiguration and
the declaration kept applying. The declaration is fixed when the bucket
is created, so the delete now returns MethodNotAllowed.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* test(s3vectors): keep an inherited WEED_S3_SSE_KEK out of the test cluster
The gateway refuses to start SSE-S3 with both s3.sse.kek and s3.sse.key
set, so a developer shell exporting WEED_S3_SSE_KEK made the AES256
bucket test fail.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* seaweed-vector: default --namespace to the gateway's Lance port
The old default, http://localhost:8888/lance, is a path nothing serves:
the Lance Namespace listens on its own port, 9101 by default on weed s3,
weed server, weed filer and weed mini. With it, the filer answered the
engine's catalog calls with a 500 and every CreateIndex failed with
InternalFailure.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* seaweed-vector: default --listen to a port no SeaweedFS server takes
127.0.0.1:8080 is the volume server's default port, so weed-vector next
to weed server exited with "Address already in use (os error 98)". 8090
is not among the defaults of master, volume, filer, S3 and its Iceberg
and Lance catalogs, IAM, WebDAV, SFTP, the message queue broker and
agent, the admin server or weed mini, nor any of their gRPC ports
(HTTP + 10000).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* lance: take any key while the S3 server runs without identities
Without identities the S3 port accepts any signature and treats the
caller as the admin. The Lance catalog still checked an x-api-key or
Bearer token against identities that did not exist, so weed-vector
started with --storage-option credentials on such a server got 401 on
every catalog call. The catalog now skips both checks when S3 auth is
off and leaves the caller to the S3 authenticator, which admits it as
the admin, as it already did for a request with no key. With
identities configured nothing changes: an unknown key is still 401.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* lance: tell clients to skip signing while S3 auth is off
A client with no credentials of its own could not reach a table's data
on a server without identities, though the S3 port takes unsigned
requests there: lance fell back to the AWS default credential chain and
failed with CredentialsNotLoaded. weed-vector without --storage-option
failed every CreateIndex that way. The catalog now adds
aws_skip_signature to the storage options it hands out while S3 auth is
off, and leaves it out once identities exist.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* seaweed-vector: bind the listener before registering with the master
The engine spawned its master registration and then bound --listen, so
an engine whose port was taken could reach the master before it exited:
on weed server with 18080 taken the master logged "vector node
127.0.0.1:18080 added" and removed it in the same millisecond, and the
gateway noticed it. Bind first, and name the address in the error
instead of "transport error".
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* seaweed-worker: keep an unsigned store off the AWS default chain
A store the catalog says takes unsigned requests needs no credentials,
but lance builds the AWS default credential chain for any store without
them, and the chain probes instance metadata on every dataset open.
With no credentials of its own, weed-vector spent a second on each
query (1030-1066 ms against 32-46 ms with --storage-option ones). The
merge of the operator's fallback and the vended options, which the
engine and the Lance worker each did in their own copy, moves into
table_options, and gives such a store empty credentials.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* test(s3vectors): serve an index with no identities and no engine credentials
A second cluster runs weed s3 without -config and weed-vector without
--storage-option, with no AWS credentials in either environment: weed s3
would turn them into an admin identity, and the engine would find them.
On master its CreateIndex fails with InternalFailure.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
A euclidean index answered QueryVectors with Lance's L2 distance, which is
the squared Euclidean distance: [3, 4] was 25 from the origin, not 5. AWS
defines the metric as the straight-line distance between the vectors. The
gateway now returns its square root, so the single engine and the enterprise
coordinator, which both hand back the raw Lance value, report the same
distance. Ordering is unchanged.
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* s3vectors: resolve the caller the way table buckets do
The vector checks used the identity name as principal, while the catalog
records a bucket's owner through getAccountID: the account id, or the
issuer-scoped user id of a federated identity. An identity with an account
or a federated identity did not own the vector bucket it created, a Deny
naming its account did not bind it, and an Admin identity in the admin
account did not get the admin bypass it has on table buckets.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* s3vectors: honor identity actions scoped to a vector bucket
An identity action such as s3vectors:QueryVectors:team-a granted nothing,
because the vector policy context never named the bucket. Table buckets
match <action>:<bucket> through the same context field, so pass the bucket
the check is about. Creating a bucket stays unscoped, as for table buckets.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* vector: build without a system protoc
lance's build scripts run $PROTOC and fail without one. Point it at the
protoc that protoc-bin-vendored ships, at the version in Cargo.lock, so
building inside seaweed-vector/ needs only the Rust toolchain. An explicit
PROTOC still wins, and a registry without the crate falls back to PATH.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* test(s3vectors): build weed-vector inside its workspace
Cargo reads seaweed-vector/.cargo/config.toml only when it runs there, so
build from that directory to pick up the vendored protoc.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* ci(vector): drop the PROTOC step the workspace config replaces
Builds run inside seaweed-vector/, where the cargo config already points
lance at the vendored protoc.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* test(s3vectors): resolve CARGO_TARGET_DIR before changing directory
A relative CARGO_TARGET_DIR given to make would point cargo, which now runs
in seaweed-vector/, and the cp after it at different directories.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* vector: pick the vendored protoc on every Linux host it ships for
protoc-bin-vendored also carries ppc64le, s390x and x86_32 binaries.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* shell: dial scrubbed volume servers on their registered gRPC port
volume.scrub, ec.scrub and volume.server.state built each data node's
address from its HTTP address alone, so a volume server whose gRPC port
is not the HTTP port + 10000 was dialed on the wrong port.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* shell: resolve -node volume servers to their registered gRPC port
An explicit -node/-nodes list was dialed on HTTP port + 10000 too.
Match each entry against the topology and take the data node's gRPC
port; an address the master does not know is used as given.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* shell: keep a gRPC port given in -node
A node given as host:port.grpcPort was rewritten to the registered gRPC
port; only a bare host:port takes it from the topology.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* shell: dial -node volume servers when the topology is unavailable
An explicit node list did not need the master before; keep using it as
given if the volume list cannot be fetched. Selecting every node still
requires the topology.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* shell: bound the topology lookup for an explicit -node list
With the master down, volumeServer.state -nodes waited on the master
forever instead of reaching the given volume servers. Give the lookup a
30s deadline and fall back to the addresses as given.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* shell: resolve volume server addresses from a topology-only listing
Only the data nodes are needed to find each server's gRPC port, so ask
the master for the topology without its volumes and ec shards. A large
cluster no longer risks running past the -node lookup deadline and
falling back to the HTTP port + 10000.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* shell: skip the topology lookup when every -node names its gRPC port
A host:port.grpcPort entry is dialed as given, so a list made only of
those does not need the master and no longer waits on it during an
outage.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* security: share the filer admin token check
CheckGrpcAdminPeer verifies an admin Bearer token inline. Move the check
into IsFilerAdminToken so an HTTP endpoint can accept the same token.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* lance: recognize the cluster's own components by a signed token
A request whose X-Seaweedfs-Cluster-Token header carries an admin token
signed with jwt.filer_signing.key comes from a component that holds the
cluster key: the vector engine or the Lance maintenance worker. Mark such
a request as trusted. It still authenticates as an identity, which stays
the principal for vended credentials. A dedicated header leaves the
Authorization header to Lance OAuth clients. A gateway with no key
trusts no one.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* lance: vend write credentials to a vector bucket's indexes only to trusted callers
An index in a vector bucket's vector namespace is also a Lance table, so
a caller granted s3tables:UpdateTable could describe it, or declare it
again, and get credentials that write the index data without passing the
vector bucket policy. In that namespace both now vend read-only
credentials unless the caller is trusted, and a declare from anyone else
no longer writes the dataset markers.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* lance worker: prove the cluster key to the Lance namespace
The gateway now vends write credentials to a vector bucket's indexes only
to callers that send a token signed with jwt.filer_signing.key. The
worker already takes that key for the admin stream; its namespace client
now mints a token from it for every catalog request.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* weed-vector: describe the index the gateway created and prove the cluster key
CreateIndex declared a table the gateway had just created, which only
worked because declare is idempotent. Describe it instead. The Lance
namespace vends write credentials to a vector bucket's indexes only to
callers that send a token signed with jwt.filer_signing.key, so take the
key with --filer-signing-key, the same flag the Lance worker has.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* lance worker: keep the cluster key out of Debug output
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* lance: answer a declare of an existing table with its own location
CreateTable succeeds for a name that already exists without looking at
the location requested, and declare kept using that location for the
dataset markers and the vended credentials. Declaring an existing table
with another table's location, such as a vector index, got credentials
to that table's data. Use the location the catalog holds, which
CreateTable now also reports for a new table.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* lance: sign cluster tokens with a key derived for the catalog
A cluster token was a filer admin token signed with jwt.filer_signing.key
and, by default, never expired. Sent to the Lance namespace on every
request, often over plain HTTP, a captured one would also authorize the
filer's IAM service. Sign it with a key derived from the cluster key for
this one use, which the admin services do not accept, and mint it with a
five minute lifetime. weed-vector drops its token lifetime flag.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* lance: follow a reloaded jwt.filer_signing.key
The Lance namespace copied the cluster key at startup. A SIGHUP reload
refreshes the gateway's filer guard but not that copy, so after a key
rotation the restarted vector engine and worker were no longer trusted.
Read the key from the guard on every check.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* iceberg: keep a vector bucket's indexes read-only
An index in a vector bucket is also a catalog table the Iceberg REST
catalog loads, and LoadTable vended credentials that could write it to
any caller granted s3tables:UpdateTable. A commit staged its metadata
file in the index directory on the same grant. Apply the vector
namespace rule to both, after the caller's own UpdateTable check.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* iceberg: confine a created table's location to its namespace before writing
Stage-create skips the registration that confines a location to the
table's namespace, yet vends write credentials for that location. A
caller allowed to create a table in any namespace of a bucket could name
another namespace's prefix, a vector index included, and get write
credentials to it. The non-staged create also wrote its first metadata
file there before the registration refused the location.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* lance: a declare that finds an existing table answers like describe
Creating a name that already exists is idempotent, so a caller allowed
only to create tables could declare someone else's table and get write
credentials to it, and clear its deregistered marker. Describe would have
vended that caller read-only credentials. A declare now looks the name up
first; when it only finds the table, writing needs UpdateTable, as it does
for describe.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* vector: compare json_array_contains elements as typed JSON values
Lance's json_array_contains compares each element's text with the value, or
with the value wrapped in quotes, so 1 matches "1", true matches "true", and
1.0 misses a stored 1. The filter swap now replaces it like json_extract and
json_exists: the value is parsed as JSON once per batch and compared with
jsonb's typed equality, and only literal arguments are swapped.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* s3vectors: match a filter value against the elements of an array
A filter never looked inside an array: {"tags": "space"} and
{"tags": {"$in": ["space"]}} both missed "tags": ["space", "classic"].
AWS documents that $eq, and so a bare value, matches an array metadata value
when any element equals the input. $eq, a bare value and each $in member now
also match through json_array_contains; $ne and $nin remain their negations.
Range operators, $exists and nested objects are unchanged.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* filer: scope a prefix JWT to the directory a zip extract writes under
Without a trailing slash, an extract request writes members beside the
named path, in its parent directory, while the JWT check only saw the
request path. A token scoped to /tenant1 could therefore extract into /
by posting to /tenant1?extract=zip. Add the extract root to the paths a
prefix-restricted token must cover.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* filer: do not keep an extracted archive under the name ".."
A multipart file name of ".." passes path.Base unchanged, so
extract.keep stored the archive as an entry literally named ".." in the
target directory, a name that local sinks resolve to the parent.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* leveldb3: keep a non-local bucket name out of the per-bucket folder
findDB splits the bucket out of a /buckets/ path and uses it as a folder
name under the store directory. The name cannot hold a slash and dot
names already go to the default DB, so this only rejects an empty or
non-local name, but it also lets static analysis see the folder cannot
escape the store directory.
* filer: clean the zip extract root before scoping and writing
A percent-encoded dot segment survives the path cleaning in front of the
mux, so a request path ending in "/" reached extractZipRoot with ".."
in it. The JWT scope check cleans the path it compares, but the members
were written under the raw root, so the two could disagree.
* leveldb3: apply the bucket folder check to bucket create and delete events
findDB already keeps dot and non-local bucket names out of the per-bucket
folders, but OnBucketCreation and OnBucketDeletion used the name as is. A
directory event for /buckets/.. opened a DB in the parent of the store
directory, and its delete removed that parent.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* leveldb3: keep a bucket delete event off the default DB
A bucket named _main maps to the default DB in findDB, so deleting
/buckets/_main closed the default DB and removed its folder, taking every
non-bucket entry with it.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* leveldb3: keep bucket names with a path separator out of the store dir
A bucket event named x/../_main passed the local-name check, and its
delete removed the default DB folder once store.dir/x existed.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* leveldb3: only reject the platform separator in a bucket name
A backslash is an ordinary file name character on Unix, and a bucket
folder named that way already exists for anyone who created one. Rejecting
it on every platform would route those entries to the default DB after an
upgrade. The slash can never appear in a bucket name here, and on Windows
the backslash is the separator, so rejecting os.PathSeparator alongside
the slash keeps the folder flat without changing Unix routing.
* leveldb3: check for an escaped bucket db before its delete event
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* s3 tier backend: bound connect and response-header waits
The tier client used the SDK default transport: a 30s dial timeout and no
limit on waiting for response headers. A blackholed endpoint cost about
two minutes per request with the SDK's retries, and an endpoint that
accepts connections but never answers held the request forever.
Dial with a 10s timeout and give up on a response whose headers do not
arrive within a minute.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* volume: stop the disk scan from waiting on a stalled tier backend
A volume server registers with the master only after every volume on its
disks has loaded, and loading a tiered volume reads its superblock from the
remote object. With the S3 endpoint unreachable, each read waited out the
client's timeouts and retries on one of the loader workers, so start-up grew
with the number of tiered volumes and the local volumes were not served
until it finished.
A tiered volume already loads without its superblock when the read fails.
Give that read 10s, and once it stalls, let the backend's other volumes skip
it for a minute instead of each waiting out the timeout. The volume reads
from the backend as before once the endpoint answers again.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* seaweed-volume: bound connect and read waits of the S3 tier client
Mirror the Go tier client: a 10s connect timeout instead of the SDK's
3.1s default, and a minute to receive a response, so an endpoint that
accepts connections but never answers cannot hold a read forever.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* seaweed-volume: stop volume loading from waiting on a stalled tier backend
Mirror the Go loader: the superblock read of a tiered volume gets 10s, and
once it stalls the backend's other volumes skip it for a minute. The
volume loads with its .vif version as before when the read fails, and
reads from the backend once the endpoint answers again.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* seaweed-volume: keep the .vif replication when a remote superblock read fails
Go applies the .vif replication to a tiered volume whether or not its
superblock read succeeded, so a volume loaded while its backend is down
still reports its replica placement. The Rust loader applied it only on
success and reported the default placement instead.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* volume: skip the .idx head repair while the tier backend is unreachable
The repair reads needle headers from the remote .dat, so after the
superblock read failed it would wait on the same backend again and hold
up the load. It runs on the next load that reaches the backend.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* seaweed-volume: skip the .idx head repair while the tier backend is unreachable
Mirrors the Go load: after the remote superblock read failed, the
repair's remote header reads would wait on the same backend again.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3 tier backend: give up a ranged read whose body stops arriving
The response-header timeout ends once headers arrive, so an endpoint
that answers and then sends no body held ReadAt, and a superblock
read's worker, forever. Cancel the request once the body makes no
progress for s3ReadTimeout.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* seaweed-volume: test that a tier read whose body stalls gives up
The SDK's stalled-stream protection already fails it, matching the Go
read's no-progress cancel.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* volume(rust): keep the S3 read timeout off upload_part
The SDK's read timeout runs from the start of the request, so a 64 MiB
part on a slow link would time out while its body is still being sent.
Go's ResponseHeaderTimeout only starts after the body is written.
upload_part now keeps the connect timeout and drops the read timeout;
stalled-stream protection still fails an upload that stops moving.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* volume(rust): bound the upload_part response wait from when the part is sent
Without a read timeout, an upload_part whose body was accepted but never
answered waited forever. The part body now goes to the connection in 64 KiB
chunks, and once the last chunk is taken the response must arrive within
S3_READ_TIMEOUT, like Go's ResponseHeaderTimeout. A retry rebuilds the body
and restarts that clock. The body keeps its in-memory contents, so signing
and the checksum header are unchanged.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* volume(rust): keep a retried part's response clock from an earlier attempt
The send state was a single flag, and a retry body could be rebuilt and
fully sent before the waiter saw the flag drop, so the waiter kept the
earlier attempt's deadline. Each body now carries its attempt number, only
the latest attempt marks the part sent, and the deadline holds only while
no later attempt has started.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* ci(release): publish weed-vector binaries
Tagged releases now carry weed-vector_linux_{amd64,arm64}.tar.gz, static
musl builds made the same way as weed-worker's.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* docker: carry weed-vector in the image
The image takes a pre-built weed-vector the way it takes weed-worker, or an
empty placeholder where none was built. `vector` runs it.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* ci(docker): build weed-vector for the images
The container workflows cross-compile weed-vector for amd64 and arm64
beside weed-worker and place it in the Docker context.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* install.sh: add the vector component
`--component vector` installs weed-vector from a release, and `all` adds it
when the release carries one, as it does for the worker.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* ci(docker): skip weed-vector for source refs that predate it
The manual latest-image rebuild accepts any source_ref; one without the
engine's server crate now builds its image without weed-vector instead of
failing the Rust job.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* docker: have the vector engine listen beyond loopback
weed-vector listens on 127.0.0.1 by default, which a gateway in another
container cannot reach. `vector` now adds --listen 0.0.0.0:8090 unless a
listen address is given by flag or WEED_VECTOR_LISTEN. The engine registers
its container address for a wildcard listen.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* localsink: skip keys that resolve outside the sink directory
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* filer: reject entry paths with dot segments
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* filer: test renames onto dot-segment targets
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* filer: keep the dot-segment rename test inside one bucket scope
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* localsink: resolve keys through an os.Root on the sink directory
The lexical check alone lets a symlinked directory that already exists
under the sink directory carry writes and deletes outside it. Create,
rename and delete now go through an os.Root opened on the sink
directory, so a symlink can only resolve inside it.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* localsink: follow absolute symlinks that resolve inside the sink directory
os.Root rejects every absolute symlink target, so a backup through an
absolute link that points back into the sink failed, and a delete through
it left the file behind. Resolve the parent of the key first and use the
resolved path when it stays under the sink directory. Operations still go
through os.Root, so a link swapped after the check cannot escape.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* operation: read a manifest chunk from the next replica when one fails
ChunkedFileReader read each chunk from one random location and returned
the first error, so a chunk whose cached replica went down failed the
whole file although another replica had it. Try the chunk's locations in
turn (local-data replicas first) and resume each attempt at the byte the
previous one reached. A failed write to the caller still ends the read.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* operation: drop a chunk volume's cached locations after a failed read
A chunk read that failed on a replica left that replica in the
10-minute location cache, so later reads kept trying it first. Drop the
volume's cached locations after a failure, and when every cached
replica has failed, ask the master once more and try the locations it
now lists.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* operation: don't continue a partly sent chunk on another replica
A volume server streams a large needle before it has verified the
checksum and aborts the response on a mismatch, so bytes that already
reached the reader may be corrupt. Continuing such a chunk on another
replica would hand back a mix of bad and good bytes as a complete read.
Move to the next replica only while nothing of the chunk has been
written; a mid-body failure still fails the read, and the dropped cache
entry lets the next read pick a live replica.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* operation: treat a short chunk body as a failed replica read
A replica that ends a chunk's body cleanly before the chunk's size made
the read look successful, so the next replica was never tried and the
following chunks landed at the wrong offsets. Read exactly the chunk's
remaining length and report a short body as an error.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* fix(s3): verify SigV4 signature with decoded path behind X-Forwarded-Prefix
When SeaweedFS is hosted behind a reverse proxy that strips a URL prefix (X-Forwarded-Prefix) and forwards the percent-encoded request path, signature verification fails for object keys containing non-ASCII characters. The client signs the canonical URI as the single percent-encoded path including the prefix, but the server verifies against the already-encoded forwarded path joined to the prefix, producing a mismatch (SignatureDoesNotMatch).
Decode the forwarded path and rejoin the prefix before verifying, mirroring the existing un-prefixed decoded retry. ASCII keys are unaffected because the decoded path equals the original, so the retry is skipped.
Signed-off-by: Mohd Quamar Tyagi <104281681+Tyagiquamar@users.noreply.github.com>
* s3: build the SigV4 candidate paths once and try them per host
The forwarded-prefix and plain branches each repeated the decode-and-verify
step for every host candidate. List the escaped and decoded paths once, the
prefixed forms first, and verify them in the same order as before.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Signed-off-by: Mohd Quamar Tyagi <104281681+Tyagiquamar@users.noreply.github.com>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* s3tables: let DeleteTableBucket remove empty leftover directories
Page through the bucket instead of reading one page, and check each bare
directory a deleted namespace left behind for files: empty ones go with the
bucket, one holding files refuses with 409 naming it.
The delete claims the bucket before scanning, and CreateNamespace refuses a
claimed bucket and confirms it again after its write, so a namespace created
meanwhile is seen by the scan or turned away instead of deleted with the bucket.
The bucket is removed only while it still carries that claim, otherwise 409.
A delete that refuses lifts the claim, retrying a failed release, since a claim
left behind refuses every create in the bucket.
The new namespace stays marked unfinished until that check passes, so no table
or view is created in one that gets backed out; a create takes over a namespace
still carrying the mark. Both checks accept only the bucket the create was
authorized against, by its creation time: one recreated under the same name
answers 404 and gets nothing written into it.
The removal is not recursive. Each leftover is claimed first, only while it is
still the directory the scan saw, and goes only while it holds no file or
catalog record; the bucket goes only while nothing is left in it. A file
uploaded or a namespace taken back after the scan is refused with 409 instead
of deleted.
* s3tables: keep directory-backed S3 objects in DeleteTableBucket
* s3tables: keep a file that replaced a leftover directory in DeleteTableBucket
* s3tables: claim only the authorized bucket in DeleteTableBucket
* s3tables: assert the prefix-object mark when claiming a leftover directory
* s3tables: hide unfinished namespaces from readers and DeleteNamespace
* s3tables: stop CreateNamespace on any failed bucket check
* s3tables: retry lifting claims on leftover directories
* s3vectors: declare the index namespace again after losing it to a concurrent declare
CreateNamespace now writes a namespace marked unfinished and confirms it
after checking the bucket. A second CreateIndex that declares the namespace
in that window takes the marked record over, and the first one's
confirmation answers 409, which failed its CreateIndex. Declare it again,
a bounded number of times, when the create answers a conflict.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3tables: name an unfinished namespace that keeps a bucket from being deleted
A CreateNamespace that dies between its write and its confirmation leaves
a namespace marked unfinished. Listings and GetNamespace hide it and
DeleteNamespace answers that it does not exist, while DeleteTableBucket
counts it, so the bucket could not be deleted and nothing said why. Name
it in the BucketNotEmpty answer; creating it again takes it over.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3tables: read a refused empty-directory delete with filer.DeleteEntryError
Use the filer's own reading of a non-recursive delete's answer instead of
a copy of its message prefix.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3vectors: release the held create when the concurrent-declare test fails
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3vectors: buffer the held create's result so it can exit after a failure
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3tables: retry lifting a refused namespace delete's claim
A claim left on a namespace refuses every create in it, the same harm the
bucket's claim does, so its release retries too. With no one-shot caller
left, the retrying release takes the releaseDeleteClaim name.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3tables: keep the data under a hidden directory on bucket delete
The scan filed every hidden entry by name for a recursive delete, so .hidden/data went with the bucket. A hidden directory waits with the leftovers instead: the walk refuses one that holds a file or an object, and removeEmptyTree takes only one that is still bare and empty. A hidden file is removed by name as before.
---------
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Chris Lu <chris.lu@gmail.com>
* lance worker: let dataset::open take an object store wrapper
open_with is open with the dataset's object store wrapped, so a caller
such as weed-vector can put a read cache in front of the table's files.
open keeps its signature and opens unwrapped, as before.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* vector: cache reads of write-once Lance files across requests
Every VectorService call opens the table afresh, and Lance keeps no
data pages between opens, so each QueryVectors re-read the whole
vector column through the S3 gateway: 83 MB and about 228 range GETs
per query at 50k x 384. The gateway and volume server spent their CPU
streaming it and throughput stayed at 6-8 queries/s whatever the
client concurrency.
weed-vector now wraps each table's object store with a bounded
in-memory cache (--read-cache-mb, 512 MiB by default, 0 disables). It
only serves data files, deletion files and index files: Lance writes
each once under a fresh random name, so a cached read cannot go stale,
even when an index is dropped and recreated at the same location.
Manifests, tags and every other path still go to the store, and each
call still describes the table, loads the latest version and takes
the query lease, so reads see every write committed before them.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* vector: cache an owned copy of each fetched read
A range from get_ranges can be a slice of a larger response buffer,
such as a coalesced fetch, and caching the slice kept the whole buffer
alive while the weigher counted only the slice. The cache now copies
each fetched range and whole-object read into its own buffer, so
--read-cache-mb bounds what it actually holds.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
The volume server views summed the logical size of the volumes into
DiskUsage while DiskCapacity came from the filesystem statfs size. On a
compressing filesystem the logical sum exceeds the physical capacity, so
the server details popup derived a negative available space and rendered
it as "NaN undefined".
Pair each disk's capacity with the usage measured the same way, using the
statfs numbers the volume server already heartbeats, as CollectTierStats
does. Volume servers without statfs numbers keep the previous slot-based
capacity and logical bytes, and EC shard bytes are only counted there
instead of on top of the filesystem usage.
* test(s3vectors): add an opt-in end-to-end perf suite
perf_test.py runs on the same cluster fixture as the integration suite
(Go or Rust volume, single or distributed queries) and times the data
plane through the S3 Vectors API: PutVectors ingest and upsert, QueryVectors
latency with and without metadata and a 10% filter, query throughput
across client threads, GetVectors, segmented ListVectors, and
DeleteVectors. Recall is checked against an exact client-side search and
listings must be complete; timings only report. Sizes come from PERF_*
variables, and results go to a JSON report and a printed summary.
It runs through run_perf.sh or `make perf`. run_test.sh does not collect
it, so CI is unchanged.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* test(s3vectors): check every listed key in the perf suite, not just the count
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* test(s3vectors): cap perf samples at PERF_VECTORS so small runs work
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* test(s3vectors): warm perf worker clients before starting throughput timers
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* test(s3vectors): report the key count of perf GetVectors and DeleteVectors calls
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* vector: parse a filter's JSONPath once per batch instead of per row
Lance's json_extract and json_exists parse their JSONPath argument again
for every row they see. Every translated S3 Vectors filter goes through
them, so a flat scan paid about 1 µs per row before the predicate
compared anything, and a 10%-selective {"category": {"$eq": "c3"}} ran
about 50% slower than the same query unfiltered.
The path is always a literal, so the scanner now parses the filter as
before and swaps in copies of the two functions that parse it once per
batch. They call the same jsonb selector and serializer, so results and
errors do not change. Calls whose path is not a literal keep Lance's
functions. The filter SQL is unchanged, so the gateway, existing tables
and the distributed workers (which run query::ann) need no change.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* s3vectors: read the field once per extractor in $in and $nin
$in and $nin rendered one IN per member, each with its own json_extract,
so an N-member list parsed the row's metadata N times. Members that share
an extractor (strings and booleans, or numbers) now go into one IN list,
so a list reads the field at most twice. Matching is unchanged: each
group still only sees its own JSON type and still coalesces NULL before
$nin negates it.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* s3vectors: rename the $in literal map so codespell passes
Codespell reads "lits" as a misspelling of "list".
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* volume: serve a ranged GET without reading the whole needle
A ranged GET read every needle up to 1MB in full, checksum and all,
then copied out the requested bytes. Read the metadata and stream only
the range instead, as the Go server already does above 1MB and the Rust
server does at any size. A small range of a 790KB needle goes from
~340us to ~160us.
The metadata read stays non-terminal: the range read that follows
reports the io outcome, so a failing disk still builds its error streak.
HEAD, compressed needles and chunk manifests keep their current reads.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* volume: read the whole needle for a range that may need an image transform
Without a URL extension, the stored name can still make a ranged GET
with width, height or crop parameters an image transform, which needs
the payload. Keep the full read there instead of streaming the range.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* volume: stream a range from the record its metadata came from
The streamed read looked the needle up again, so a replacement written
after the metadata read sent new bytes under the old length, checksum
and headers. Read the record the metadata read resolved, as the Rust
server does, and after a compaction refuse a record whose size changed.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* rust volume: read the whole needle for a range that may need an image transform
A ranged GET with no URL extension but width, height or crop parameters
took the metadata-first path, so a stored image name never got its
transform. Read a small needle whole there, as the Go server does.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* volume: keep later parts of a multi-range reply on a moved record
When a compaction moved the record before one part of a multi-range
reply, that part re-resolved it, but the next part went back to the
offset the metadata read saw. Remember the re-resolved offset too.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* volume: check a compaction-moved record is the one the metadata came from
After a compaction, a range re-resolved the needle and accepted any record
of the same size, so a replacement written in between could send its
bytes under the old headers. Compare the moved record's checksum and
append time with the metadata already read, and fail otherwise.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* volume: accept a re-encoded checksum and count read errors on a moved record
Compaction rewrites a pre-3.09 checksum in the current encoding, so the
moved-record check now accepts either. A failed read of the moved
record's metadata now feeds the io-error streak and surfaces as that
error instead of not-found. The pinned record is a fresh value rather
than a write through the index's pointer.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* volume: keep a moved record's wrapped offset when a range follows it
With 4-byte offsets, a record past the 32GB wrap is read at offset plus
MaxPossibleVolumeSize. The compaction refresh dropped that, so it checked
and streamed the wrong location. Retry the wrapped offset in the moved
record check, as the metadata read does, and apply it to the stream.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* vector: scan only the fragments a ListVectors page covers
A page pushed its offset into the scan, but the scan still loaded every
fragment first, reading each one's deletion file: about 105 S3 GETs per
page at 50k rows against 25 at 10k. The page's fragments now come from the
live row counts in the manifest, so a page reads only its own. Tokens and
row order are unchanged; a manifest without the counts keeps the old scan.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* s3vectors: list a ListVectors segment in the engine
The gateway paged through the whole index for every segment and kept one
row in segmentCount, so a segment cost segmentCount engine pages and 16
segments listed slower than one. The engine now takes segment_count and
segment_index: it reads keys across the stride, keeps the segment's rows
and takes their data and metadata alone, one call per page.
A segment is still the rows whose position modulo segmentCount is
segmentIndex, and its token still the position to resume from. An engine
that predates the fields does not set segmented, and the gateway then
pages through it as before.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* vector: read only the key column across the table in GetVectors
The key lookup scanned every column. On S3, lance reads a column during
the scan when a row of it is under 1000 bytes, so a 128-dimension vector
was read for every row of the table to return 100 of them, and data and
metadata were read even when the request did not ask for them. The scan
now projects the requested columns and materializes everything but the
key late, for the matching rows only.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* s3vectors: start the legacy segment fallback from the page already read
With an engine that ignores segment_count, the gateway dropped the page
it had just read and fetched it again. The fallback now consumes that
page first, so a one-row segment page still costs one engine call.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* vector: check metadata on segmented ListVectors pages
The segment test listed data only. Its rows now carry metadata on two in
three keys, and every segmented row, read through take_rows, must equal
the same row in a plain listing.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* vector: lease the version while a segment takes its data
A segmented page with data or metadata reads keys first and then takes
the other columns from the same version, so maintenance could reclaim
its files in between. That path now holds a lease like GetVectors. A
keys-only page is one scan and stays unleased, as plain pages are.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* vector: leave GetVectors materialization to lance
Forcing every column late made a 100-key GetVectors with data slower at
10k x 128 in the perf cluster: 54.7 ms against 46.3 ms. The take issued
a read per row, where lance's heuristic reads a narrow vector column in
one pass per fragment. The projection keeps the gain on its own: with
neither data nor metadata requested, the call went from 38.6 ms to
25.3 ms and from 150 to 89 S3 GETs. At 50k x 384 the vector is already
late under the heuristic, and a full lookup is unchanged.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* vector: follow a leader hint at once and skip one that failed this pass
A follower naming a dead leader stalled master failover. Each pass broke
off at the first hint, tried that leader first on the next pass, then
got the same hint from the same follower, so the later configured
masters were never tried. A pass now works like wdclient.tryAllMasters:
it follows a hint at once, does not follow a hint to a master that
already failed in that pass, and moves on to the rest. The hops are
bounded by the master count, as the volume heartbeat bounds its hints,
so two masters naming each other cannot spin.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* vector: test a hint naming a failed master in its gRPC form
A pass keys failed masters by host:http, so a configured m1:9333 that
failed must also stop a hint of m1:9333.19333. Cover that equivalence.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* vector: refuse to drop a location that is not the table's own
DropIndex removed everything under whatever location the catalog
described. That location is caller-supplied through the Lance
namespace and only confined to the bucket, so a table registered at
s3://bkt/vector, or at another index's directory, wiped every index in
the bucket or that index. drop now accepts only the colocated path the
gateway's tableLocation derives from the table id, after trimming a
trailing slash, and the RPC answers FailedPrecondition otherwise. This
mirrors the ancestor and live-entry guards on the catalog's DeleteTable.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* vector: bound the master dial and try a leader hint once
A remembered leader hint was never cleared and the dial had no connect
timeout, so a black-holed former leader added the kernel's SYN timeout
(about two minutes) to every reconnect pass before the configured
masters were tried, and registration lapsed. The dial now gives up
after 5s, the same bound wdclient puts on a master call, and a hint is
tried once ahead of the configured masters, as wdclient.tryAllMasters
keeps nextHintedLeader for one cycle only.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* vector: refuse a master port whose gRPC port would pass 65535
grpc_addr derived the gRPC port as http + 10000 in u16, which panics in
debug builds and in release silently dials a wrapped port for HTTP
ports above 55535. Go's pb.ServerToGrpcAddress adds in int, so the port
never wraps and the dial fails instead. The addition is now checked,
and weed-vector refuses to start with an error asking for the explicit
host:http.grpc form.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* vector: refuse a numeric master port past 65535
A master given as m1:65536 or m1:9333.65536 failed both u16 parses and
fell through to the raw address, so startup accepted it and every dial
failed. Ports now parse as u64, as Go's ServerToGrpcAddress does with
ParseUint, and one outside u16 is refused at startup.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* vector: bound the master's first keep-connected answer
The connect timeout only covers TCP. A master that accepts the stream
and never answers held stream_to forever: grpc-go sends the response
headers with the first message, so the call never returned and later
seeds were never tried. A healthy master always answers at once (a
follower with its leader hint), so the stream open and the first
response now share the same 5s bound; after that the long-lived stream
stays untimed.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
s3tables: keep the fence on a declare of an existing index
Deciding the existing-index exception before the handler ran left a
window: an index deleted between that lookup and the handler's own check
would be recreated, unfenced, by a caller without s3vectors:CreateIndex.
Drop the up-front exception, so every CreateTable runs fenced and a
fresh create is refused at its first write. The engine's declare still
passes because the idempotent path writes nothing; for a build whose
idempotent path re-applies state already in place, the fence answers an
UpdateEntry that leaves the entry exactly as stored without writing.
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* s3tables: make the vector namespace of a vector bucket read-only
S3 Vectors authorizes index operations against the vector bucket policy,
but an index is also a catalog table in the bucket's "vector" namespace,
and the S3 Tables API only checks s3tables grants. A caller the vector
policy denies DeleteIndex could still delete the index, and its data,
with DeleteTable.
Untrusted writes that name the vector namespace of a LANCE bucket now run
behind a fence that passes filer reads and refuses filer writes. The
operation's own checks still answer everything short of a write, so a
caller without the grant gets its usual denial and learns nothing about
the bucket, and a CreateTable that finds the index already there, which
is what the vector engine's declare does, still succeeds. A refused write
answers 403 AccessDenied. Trusted callers (the gateway, the shell) are
unaffected, and so is a namespace named vector in an Iceberg bucket.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* lance: refuse register and deregister in a vector bucket's vector namespace
Every catalog write the Lance namespace makes goes through the S3 Tables
handler and its vector namespace fence, but register over an existing
table at the same location and deregister only add or remove a marker
file in the dataset directory. Check the vector namespace before those
writes, after the table read that already gates them.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3tables: refuse deferred creates in a vector bucket's vector namespace
Manager.AuthorizeCreateTable gates the Iceberg create paths that write
metadata into the table bucket before the catalog registers the table.
Those writes never pass the handler's fence, so the gate applies the same
read-only rule once the caller's own create permission is confirmed.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3tables: let a declare of an existing index past the vector namespace fence
The vector engine declares every index the gateway has just created, and
that CreateTable only reaches its idempotent path. Upstream that path
writes nothing, so the fence never trips, but a build whose idempotent
path repairs table state would refuse the engine. Decide the exception
up front instead: a CreateTable naming a finished index in the vector
namespace runs unfenced, while a new name or an unfinished create stays
fenced.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* vector: throttle worker discovery from lookup end
cli() set nextResolve before resolve() ran, so a lookup that takes as
long as the throttle interval (an unreachable first master costs the
full 5s dial timeout) left the next lookup already due when it
returned. Discovery then ran back to back and, since it holds d.mu,
every vector request queued behind it.
resolve() now sets nextResolve when the lookup lands, using the short
recheck window only while no client is cached.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* vector: resolve workers outside the discovery lock
resolve() held d.mu across the master lookup, so while it waited out an
unreachable master every vector request blocked on cli(), even with a
healthy worker already cached.
The lookup now runs in a goroutine without d.mu, and d.resolving keeps
it to one at a time. Callers with a cached client use it while the
lookup is in flight; callers without one wait for that lookup and share
its result. Throttling, the sticky worker and reset(c) are unchanged.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* vector: keep the cached worker connection when a redial fails
resolve() closed d.conn before dialing the replacement worker. A
synchronous dial error then left d.client bound to a closed connection.
Requests got codes.Canceled, which never triggers reset, and later
refreshes saw the unchanged d.worker and never redialed. The old
connection is now closed only after the new dial succeeds.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* s3vectors: bound ListVectors maxResults and segment parameters
segmentCount had no upper bound, so segmentCount=1000000,
segmentIndex=999999 made the segment emulation scan the whole index to
return nothing. Apply the AWS ListVectors windows before authorization
and any engine call: maxResults 1..1000, segmentCount 1..16,
segmentIndex in [0, segmentCount), and segmentCount and segmentIndex
given together. The fields become pointers so an explicit 0 is rejected
while an omitted one still takes the default.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3vectors: store and return distanceMetric in lowercase
CreateIndex stored distanceMetric exactly as sent, so an index created
with "COSINE" came back as "COSINE" from GetIndex and QueryVectors.
AWS returns the lowercase enum values. Lowercase the metric when the
index is created, and again when the record is read so indexes already
stored in mixed case render the same.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* s3vectors: authorize CreateVectorBucket as an s3vectors action
CreateVectorBucket went through the catalog's CreateTableBucket check, so
an identity granted s3vectors:* was denied the create while s3tables:*
was allowed it. Check s3vectors:CreateVectorBucket in the gateway, the
same way the catalog checks a bucket create, and create the record on
the internal manager under the caller's name so the caller stays the
owner.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3vectors: delete only the index record DeleteIndex checked
DeleteIndex read the record, checked it is a Lance index, then deleted
by name. A table that replaced the index in between was deleted too.
Send the checked record's version token, so a replaced record fails the
delete with a 409 ConflictException.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3tables: reject s3vectors rules in any letter case
validatePolicyDocument kept s3vectors actions and resources out of the
shared policy slot with case-sensitive checks, so "S3Vectors:GetIndex"
was stored as a rule that never takes effect. IAM service prefixes are
case-insensitive; fold case in both checks.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* ec.balance: spread concurrent shard moves over target nodes
The planner gives every freshly encoded volume the same target order, so
running each volume's moves in plan order sends all concurrent copies to the
same node while the other nodes idle. Pick, among the volumes not already
moving, the pending move whose target receives the fewest moves right now.
One volume still never has two moves at once, and moves of one shard keep
their plan order.
* ec.balance: let a move into a node wait for the move of its volume leaving it
The planner counts the slot an outgoing shard frees as room for a later
incoming shard of the same volume, and the receiver refuses a shard it has no
slot for. Make the spread tests deterministic: hold the first moves until all
have started, and check that errors of several volumes are all returned.
* ec.balance: pin that the spread moves run concurrently and in plan order without parallelism
* wdclient: publish applied volume-location deletes to registered listeners
The master already tells each connected client which volumes a server
stopped hosting; only the MasterClient's own vidMap applied it. Expose
the same filtered deletions (moves between a server's disks excluded)
through OnVolumeLocationDelete so clients keeping a second location
cache can drop the retired server too.
* wdclient: drop retired volume servers from the FilerClient location cache
The S3 gateway's FilerClient keeps its own vidMap, filled by filer
lookups, and only forgot a retired volume server after a read against
it failed (a ~15s header-timeout stall when the host still resolves).
When built with a MasterClient, subscribe to its applied deletions so
the cache forgets the server as soon as the master reports it gone.
* wdclient: drop retired EC servers from the regular location map too
Filer lookups cache EC locations alongside regular ones in vid2Locations,
so an EC removal has to clear the retired server there as well as in the
EC map for the delete to reach what reads actually use.
* wdclient: tombstone pushed deletions against in-flight provider fills
A filer lookup already running when a delete lands can return the retired
location afterwards and re-cache it. Keep a short tombstone on each
deleted (vid, url) pair so fills neither cache nor return it while the
lookup source is still settling.
* wdclient: clear the FilerClient location cache on master reconnect
Removals the master pushed while the gateway link was down never arrive
as deletes, so on reconnect the lookup-fed cache is dropped outright
rather than kept generation-wise like the stream-fed one, and relearned
through filer lookups as before.
* wdclient: invoke push hooks outside their registration locks
A callback that registers or replaces a hook would wait on the read lock
the invocation already holds and stall the master stream. Copy the hook
under the lock and call it after the lock is released.
* wdclient: report pushed adds so confirmed re-adds clear the suppression
A server coming back at the same URL inside the tombstone window stayed
hidden from provider fills. updateVidMap now carries the additions it
applied through the same hook (renamed OnVolumeLocationChange), and the
registered cache untombstones them without eagerly caching them.
* wdclient: amortize tombstone expiry over a delete batch
A server leaving takes every volume it hosted with it, and scanning all
recentDeletes per removed vid squared the batch. Prune lazily on reads
and at most once per half TTL on writes.
* wdclient: let every location cache subscribe to pushed volume changes
The setters were single-slot, so a second FilerClient built on the same
MasterClient would have silently replaced the first one's subscription.
Registration now returns an unhook func, delivered to all listeners, and
FilerClient.Close releases its own.
* wdclient: hold the deletion fence until the master re-confirms a location
* wdclient: check the deletion fence and the cache add under one lock
* wdclient: drop provider answers that crossed a cache reset
* wdclient: bound the deletion fence by evicting its oldest mark
* wdclient: refresh a repeated removal instead of evicting another mark
* wdclient: evict the oldest deletion-fence mark in constant time
* wdclient: fence a server only once it serves the volume in no form
ec.encode registers the shards before the regular copy is deleted from the
same server, and ec.decode mounts the regular copy before the shards come
off it. Both arrive add-first in separate heartbeats, so the later removal
tombstoned a (vid, url) pair the server still serves and dropped it from
the FilerClient map; the filer's answer naming that server was then fenced
out and the volume read as not found until the gateway reconnected.
Tombstone and report a removal only when neither map still names the url
for that volume.
* wdclient: serve a provider answer that crossed a cache reset
The answer is as fresh as the provider's view; only the cache it would
land in has been re-told. Dropping it failed every read whose lookup was
in flight during a master reconnect with 'volume not found'. Return it
and leave the new view to be filled by the next lookup.
* wdclient: test the moved volume whose old server keeps proxying
* wdclient: stop caching inside the master lookup provider
The provider wrote every answered location through addLocation, which
clears the deletion fence: a lookup still naming a server a pushed delete
had just retired untombstoned and re-cached it. Fallback lookups already
gate caching through fillLocation, and authoritative lookups bypass the
cache by contract, so the provider no longer writes the map at all.
* wdclient: count only master-pushed locations when suppressing a removal
hasLocation could not tell a confirmed regular copy from a lookup fill:
the master answers an EC volume with its shard servers, so the regular map
holds shard locations that arrived by lookup, and entries an earlier
generation learned linger past a reconnect. An EC removal whose url still
showed either was suppressed, leaving the retired server served.
The vid map now records the generation each regular location was pushed
at, drops an unconfirmed regular entry alongside the shard that fed it,
and answers hasServingLocation from pushed marks and current-generation
entries only.
* wdclient: test removals against lookup-cached and stale locations
* wdclient: fence removals only in the gateway cache
The fence and the pushed-location marks ran inside every MasterClient,
including the filer's, whose map is also filled by lookups and cleared
by InvalidateCache. A removal reported while the server still served the
volume (e.g. a decoded copy whose mark a failed read had wiped) left the
pair fenced until the master pushed it again, with reads failing as
volume not found meanwhile.
Only the S3 gateway's FilerClient now tombstones, from the removals its
push-only MasterClient reports. That client decides a removal from the
current generation of both its maps after the whole message is applied,
so pushedLocations goes away, and the master lookup provider caches as
before. The hooks follow the existing single-slot SetOn...Fn pattern and
are wired next to SetOnPeerUpdateFn.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* wdclient: let a removal fence expire after a minute
The master drops a broadcast when a client's channel is full, so the
re-add that would lift a fence can be lost while the gateway stays
connected. A mark held until that re-add then hid a live server, and a
volume whose only copy it held read as not found until the next master
reconnect. Bounding marks by a minute still outlasts a lookup in flight
when the removal landed; expiring early only falls back to learning by a
failed read, as before this change.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* s3 sink: forward user metadata to the destination object
filer.backup and filer.sync to an S3 sink copied the object body and tags
but dropped the X-Amz-Meta- attributes the S3 gateway stores with each
object, so the destination object came back without its user metadata.
Send them as the upload's metadata, keyed by the lowercased suffix.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3 sink: send mtime as metadata without writing it into the entry
The sink stored X-Amz-Meta-Mtime into the replicated entry's Extended map,
modifying the caller's entry, and since tags were limited to X-Amz-Tagging-
keys it was never sent anywhere. Add it to the upload metadata instead,
keeping an mtime the source object already carries.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3 sink: RFC 2047 encode metadata values that are not printable ASCII
The S3 gateway keeps user metadata values as raw bytes. Forwarded as is, a
UTF-8 value fails SigV4 on AWS and a control character makes the HTTP client
reject the request, and filer.backup retries such an object forever. Encode
those values as RFC 2047 encoded words, which S3 decodes on write.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3 sink: upload without user metadata past the S3 2 KB limit
The S3 gateway does not cap user metadata, but S3 rejects more than 2 KB of
it, and filer.backup would retry that object forever. Upload such an object
without its metadata and log a warning, as the sink did before metadata was
forwarded.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3 sink: skip metadata keys that are not valid header names
Filer gRPC clients can store any Extended key. An X-Amz-Meta- key with a
space or control character makes the HTTP client reject the whole upload,
which filer.backup retries forever. Leave such keys out with a warning.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3 sink: pick the same value for metadata keys that differ only in case
An entry holding both X-Amz-Meta-Project and the legacy x-amz-meta-project
maps both to project, and map order decided which value was uploaded. Keep
the lowest key, so the canonical form wins over the legacy lowercase one.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3 sink: use a test value codespell accepts
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* master: undo the stored record when a delete report arrives
Deleting a volume decremented the active count from the delete report's
ReadOnly flag, which the volume server never sets, so a volume the master
had already stopped counting as writable was uncounted a second time.
Every volume.move does this to its source, and the node, rack, DC and
topology counts drifted negative; the full-list path already used the
stored record. Undo exactly what was counted for the record held.
* volume: say whether a deleted volume was read-only
Explicit deletes, unmounts, collection deletes and the departures named
in a delta heartbeat all left ReadOnly unset, so an older master undid
the writable count for a volume it was no longer counting. Carry the
flags, as the Rust server already does.
* ci: share the musl Rust target setup as a composite action
Three container workflows carried the same toolchain, musl-tools and
aws-lc-sys link shim block verbatim.
* release: ship weed-worker as a static musl binary
The gnu build inherits the runner's glibc, 2.39 since the move to
ubuntu-24.04, so the released worker failed to start on Ubuntu 22.04
with 'GLIBC_2.38 not found'. The container images already build it
against musl with the same recipe.
* release: ship the Linux weed-volume as a static musl binary
Same glibc trap as the worker. The arm64 job also installed libssl,
which nothing in the volume links.
* ci: resolve the manual latest build's actions from master
Its checkout is an arbitrary source ref that may predate a local
action; the workflow itself always runs from master.
* filer: resume a rejoining peer's meta subscription at its last position
OnPeerUpdate starts every peer add at the update's arrival time. For
filers sharing one store there is no persisted offset to correct it, so a
peer that drops out and rejoins (a graceful restart) resubscribes from
"now" and the events it logged while its stream was down never reach the
aggregated buffer - filer.backup and other meta subscribers silently
skip them.
Track each peer's delivered-through position (the same timestamp domain
as the stream's events and idle-heartbeat claims) and rewind the start
of a rejoining subscription to it. First-seen peers keep the given
start, and non-shared peers still resume from their persisted offset.
* filer: keep the re-add resume point behind a replay-floored event
The session's replay floor already pins the persisted offset and the
reconnect resume point so an event that outlived its retry budget is
redelivered on resubscribe; cap peerResumeTsNs to the same position on
session exit so the new re-add resume cannot skip past it either.
* filer: stop test aggregators' log-buffer workers
* filer: cover peer rejoin resubscribing at the resume position
* filer: drop the unreachable replay-floor cap on the resume point
Only a peer with its own store can floor an event, and that peer's
subscribe start always comes from its persisted offset, never from the
re-add resume point. The cap also ran in the old session's defer, after
the re-add had already read the value.
* filer: say why a re-add may resume from an idle heartbeat
The heartbeat comments promised a reconnect never resumes from one,
while the re-add resume point does. Spell out the difference and the
fence that makes the local-stream claim safe to resume from.
* filer: forget a removed peer's resume point after an hour
The map held an entry for every address ever seen; with pod IPs that
is one per restart. The removal mark now outlives the watermark grace
and takes the resume point with it once the retention passes.
* filer: advance the resume point only for tracked peers
Same rule as the watermarks: a straggler from a dropped session must
not recreate the entry.
* filer: set the test aggregator's subscribe loop instead of a nil fallback
* fix(s3tables): keep the data under a deleted namespace
A rename leaves the source table directory holding data the renamed table
still points at, and DeleteNamespace removed it recursively. The delete now
keeps the directory, minus its catalog record, when files remain under it;
CreateNamespace can take it back.
* fix(s3tables): keep hidden-directory data, report name takeover as 409
- DeleteNamespace also looks for files under hidden directories, since a
table location may sit under one.
- CreateNamespace answers 409 when a file takes the name during the
take-back of a kept directory.
* s3tables: a vanished leftover holds nothing to keep
A bare directory removed between the namespace listing and the walk
made dirHoldsFiles answer its ListEntries NotFound, and the delete
reported that miss as a 404 or 500 on a live namespace. Skip the gone
directory instead: whatever it held is already gone, so it cannot keep
the namespace.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* s3tables: a delete's gone entry is a conflict, not a success
The filer answers a precondition delete on an entry that is already gone with FailedPrecondition carrying the not-found text, which DoRemoveRequest reads as a completed delete -- DeleteNamespace answered 200 for a delete that never ran. deleteDirectoryIfMatches now calls DeleteEntry directly, keeping the FailedPrecondition-to-conflict mapping and refusing a response-level error. keepAsLeftover likewise reports a directory removed since the scan as ErrConcurrentUpdate, so the handler answers 409 rather than 500.
---------
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: Chris Lu <chris.lu@gmail.com>
* s3: allow POST Object uploads when auth is off
With no identities configured every other handler skips authentication,
but PostPolicyBucketHandler still verified the form signature, so a
browser form upload failed with InvalidAccessKeyId (or a malformed
credential error without signature fields). Skip the signature check
when auth is off; the policy conditions are still enforced.
* s3: update AuthPostPolicy comment for the auth-off path
---------
Co-authored-by: Chris Lu <chris.lu@gmail.com>
* s3: take ListObjects marker/start-after as raw keys
Under encoding-type=url AWS url-encodes the response fields but treats
request markers as raw keys: SDKs decode Marker/NextMarker/StartAfter and
send the decoded key back. Unescaping the marker server-side read a literal
"%XX" key as the key it encodes — StartAfter=a%2Bb resumed after "a+b",
skipping it, and a paginator whose marker decoded to an earlier key got the
same NextMarker twice and failed.
Take the request value as-is; only the server's own base64: continuation
token still decodes. The marker-echo dedup now compares contents in the
response wire encoding and NextMarker in its raw form, since a raw "%XX"
marker equals the escaped emission of a different key.
* s3: take ListMultipartUploads key-marker as a raw key
Same raw-marker rule as ListObjects: under encoding-type=url the emitted
key markers are url-escaped keys, and a client decodes them before sending
the marker back, so the request value is already the raw key. Keeping the
token decode only for the non-url path, where the response could not carry
the raw key and falls back to the base64: token form.
* s3: cover ListMultipartUploads url key-marker at the handler
The selectMultipartUploads tests bypass the request marker decoding, so a
regression that unescapes an encoding-type=url key-marker would pass them.
Drive a listing through ListMultipartUploadsHandler with a literal
"a%2Bb" marker and assert it resumes after that exact key.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
A rule without MaxAgeSeconds was stored as 0, so GetBucketCors returned
MaxAgeSeconds 0 and preflights sent Access-Control-Max-Age: 0. Store -1
for unset, which corsRuleFromProto already reads back as nil.
getObjectResources and getBucketMultipartResources dropped the strconv
error, so a value that does not parse was read as 0. ListParts with
max-parts=abc then reached the filer with limit 0, which the filer reads
as its default listing limit, and returned every part with
<MaxParts>0</MaxParts>; part-number-marker=abc restarted the listing
from part 1; and ListMultipartUploads with max-uploads=abc answered an
empty, untruncated list even with uploads in progress.
Return the existing InvalidArgument errors (ErrInvalidMaxParts,
ErrInvalidPartNumberMarker, ErrInvalidMaxUploads) instead, the way
max-keys is validated. max-parts and max-uploads are parsed as 32-bit
integers to match the documented 0..2147483647 range.
* fix(volume): reject a negative suffix length in Range headers
parseRange accepted "bytes=--5" as a suffix range of -5 bytes and
returned {start: size+5, length: -5}. The volume server then answered
206 with an empty body and Content-Range "bytes 15-9/10", and the
filer's inline-content reader sliced entry.Content[15:10] and panicked.
Restore the check Go's net/http parseRange has: a suffix-length must be
a non-negative integer (RFC 7233 2.1), so a leading '-' or a negative
value is an invalid range. Apply the same check to the Rust volume
server's parse_range_header.
* test: negative-suffix manifest read is now rejected with 416
The parser change makes a negative suffix an invalid-range error, so
the duplicate-range case in this test can no longer reach the "empty
200" path. Assert the new rejection and that nothing is read.
* volume: treat a zero-length suffix range as unsatisfiable
"bytes=-0" resolved to {start: size, length: 0}, and the range handler
then answered 206 with "Content-Range: bytes 10-9/10", a value clients
must not recombine (and a first-byte-pos RFC 9110 cannot express).
Classify the zero-length suffix with the existing no-overlap case —
its resolved first-byte-pos sits at the content size — so the request
gets a 416 "bytes */size" unless another range in the set overlaps.
Applies to the Go and Rust volume servers alike.
---------
Co-authored-by: chrislusf <chrislusf@gmail.com>
Co-authored-by: user.email <chrislusf@users.noreply.github.com>
* s3: auto-create bucket on POST Object uploads to a missing bucket
PutObject and CreateMultipartUpload lazily create the bucket, but a
signed POST Object upload answered NoSuchBucket. Mirror the PUT path:
when the versioning-state lookup reports the bucket missing, run the
shared handleAutoCreateBucket gate (bucket-name check, AutoCreateBucket
option, admin identity) and then re-read the versioning state. The POST
handler already records the verified signer identity in the request
context, so the admin check and bucket-owner attribution work unchanged.
* test: assert the POST auto-create upload lands, not just non-404
The missing-bucket POST test passed whenever the handler created the
bucket, even if the upload then failed. Drive the full path through the
existing putToFiler fakes — assigned volume accepts the chunk, object
entry and payload land — and assert the 201 success response.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* s3: record verified post-policy signer's account id
The POST policy path authenticates in the handler, after the internal
account header was already scrubbed, so auto-created buckets and the
uploaded objects stored no canonical owner.
* test: assert POST auto-create records bucket owner, keep writes applied in denied case
---------
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* s3: let bucket-scoped Admin grants auto-create matching buckets
autoCreateBucket required a global Admin identity, so an identity
scoped to a prefix like "Admin:outline*" could explicitly CreateBucket
but was denied when an upload auto-created the same bucket. Gate on
CanDo(ACTION_ADMIN, bucket) — the same check explicit CreateBucket
enforces — so a prefix-scoped key lazily creates only matching buckets.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* s3: run auto-create bucket admin through the policy-aware path
A scoped Admin identity with an attached IAM policy that denied
s3:CreateBucket could still auto-create a matching bucket, because
userCanAdminBucket only consulted legacy Actions. Evaluate the same
VerifyActionPermission explicit CreateBucket gets, over a synthesized
PUT /bucket request so the attached-policy check resolves to
s3:CreateBucket on the bucket ARN regardless of the upload's shape.
* s3: record verified identity as owner of auto-created buckets
An admin authenticated outside the middleware (IAM integration, post
policy signature) left no identity in the request context, so an
auto-created bucket stored no owner and never reached the owner index.
---------
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* volume: after losing the leader, dial the leader the other masters name
When the heartbeat stream to the leader breaks and that master stays
unreachable (partitioned or powered off, so dials hang rather than being
refused), the volume server could take close to a minute to register with
the new leader, and the new leader answered assigns with "no volume server
capacity registered yet" in the meantime.
Two things in the retry path added up:
- refreshLeaderFromPeers asked the seed masters in order with up to 5s
each, inside a budget of one pulse (5s by default). With the lost
master listed first, the whole budget went to it and the live peers
that already knew the new leader were never asked.
- Even when the refresh found the new leader, the loop moved on to the
next seed master. When the lost leader had been reached through a
redirect, the next seed can be that same master, so the loop dialed it
again and waited out the 20s gRPC connect timeout.
Ask the lost master last, and when the refresh names a leader other than
the lost master, dial it next.
Assisted-by: Claude <noreply@anthropic.com>
* rust volume: dial the leader the other masters name after losing one
Mirror the Go heartbeat change: refresh_leader_from_peers asks the lost
master last and returns the leader it found, and the loop dials that
leader next instead of a seed that can be the unreachable master itself.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* rust volume: let a named leader consume the next seed slot
Restarting the seed scan on every redirect let an earlier seed that keeps
reporting the dead leader starve every seed after it: the loop bounced
between the redirect target and the next seed forever. Go consumes the
next range value for the redirect instead, so a pass always reaches every
seed. Drop the breaks so new_leader overrides the next slot the same way.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* volume: try a named leader on top of the seed scan
A reported leader used to replace the next seed slot: with seeds A,B
and A persistently naming a dead leader X, X consumed B's slot on
every pass so B was never dialed. The hint is now a one-shot attempt
made before the seed it would have displaced. Its own result stays
pending when a seed attempt names nothing, so a live follower's
redirect chain is followed while every seed is still dialed on every
pass. Mirrors between the Go and Rust volume servers.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* volume: follow a fresh leader hint before the pending seed
A named leader whose stream breaks can leave the peer refresh naming its
replacement; the loop then dialed the pending seed first and paid a dead
master's connect timeout ahead of the fresh name. Fresh hints are now
followed right away, bounded by the seed count and stopping once a name
repeats, so a peer that keeps naming the same dead leader still cannot
starve the seed scan. Go and Rust mirrored.
* volume: dial each named leader once per pass (Go + Rust)
A name that stays pending after the hint chain — the cap was hit, or the
refresh named one already dialed — was retried before every later seed in
the pass: each seed reset the hint budget, so a dead leader still cost a
sleep and a connect timeout per seed. Track the names dialed in a pass,
seeds included, and skip a hint that repeats one.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* volume: let a seed's named leader replace a pending hint (Go + Rust)
The seed dial runs after the hint chain, so the leader it names is the
freshest answer the pass has — but it was dropped whenever a stale hint
was still pending, and the next seed then chased the older name first.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* volume: keep an untried leader hint over the seed's stale answer (Go + Rust)
A seed that keeps naming a stale leader spends the per-pass hint budget
on it; its own answer then overwrote the live name the hint chain had
just reached, so that leader was never dialed. A pending hint now
survives the seed dial until it has been tried.
---------
Co-authored-by: user.email <chrislusf@users.noreply.github.com>
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* security: fail closed on misconfigured gRPC TLS
LoadServerTLS/LoadClientTLS returned plaintext credentials on every
error path once TLS was configured: a typo in tls.min_version or
tls.cipher_suites, cert and key in different directories, or cert/key
set without grpc.ca all left the component serving or dialing
plaintext. A partially configured TLS setup now stops the process at
startup instead; leaving cert/key unset still means no TLS, as before.
The loaders also stat cert/key/ca files up front since the pemfile
providers read lazily and would otherwise only fail at handshake time.
Mirrored in the Rust volume server: partial grpc.volume cert/key, a
missing grpc.ca, or a rustls build failure now abort startup rather
than warn-and-continue on plaintext.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* security: optional server-identity allowlist for gRPC clients
The mTLS client accepts any certificate the configured CA signs, so a
per-workload CA lets one component stand in for any other: a filer
identity can answer as the master. The server side already screens
callers via allowed_commonNames and allowed_wildcard_domain; nothing
screened the server.
LoadClientTLS now wires AdditionalPeerVerification when the caller
configures a server allowlist: grpc.<component>.allowed_server_commonNames
(falling back to a shared grpc.allowed_server_commonNames) matched
against the peer cert's subject CN, DNS SANs, or URI SANs such as a
SPIFFE id, plus a grpc.allowed_server_wildcard_domain suffix match.
Leaving them unset keeps the previous behavior.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* security: URI SANs are exact-match only in the server identity allowlist
A URI identity whose path happens to end in the wildcard domain would
otherwise pass the DNS suffix check.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* security: tighten gRPC TLS fail-closed and server-identity paths
- URI SANs match the server-identity allowlist exactly; the wildcard
domain suffix check applies only to CN and DNS SANs.
- A partially set client_cert/client_key pair is an error when the
cert/key fallback is empty, instead of dialing plaintext.
- Cert/key/ca files are loaded and parsed at startup (LoadX509KeyPair
and CA PEM validation) since pemfile providers read lazily.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* rust volume: error on a partial grpc client_cert/client_key pair
Mirror the Go loadClientTLS path: when client_cert or client_key is
set without its mate and the cert/key fallback is empty, fail instead
of dialing plaintext.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* security: wildcard domain match requires a valid hostname
A CA-signed identity like ".example.com" satisfied the bare suffix check
for wildcard ".example.com" despite not being a hostname. Require a real
subdomain: non-empty label before the suffix, a '.' boundary when the
configured wildcard has no leading dot, and DNS-name syntax. A leading
"*." label still matches so wildcard certs keep working. Applied to both
the Go server/client authenticators and the Rust volume's client-cert
verifier.
* security: ignore a trailing root dot in wildcard domain matching
* tls: reject a partial dedicated client cert/key pair instead of falling back
When exactly one of component.client_cert/client_key is set, warn-and-
fallback silently dialed with the serving identity. Fail closed like the
other TLS misconfigurations in this change; mirrored in the Rust loader.
Regression tests cover a partial pair with a valid fallback.
* rust volume: harden partial client cert pair test with real pem files
* security: verify dialed server identity per target kind on the filer
The filer reused one grpc.filer-verifying dial option for master, volume,
filer and s3 targets, so a broad allowlist accepted the wrong component at
the wrong port and a narrow one rejected legitimate peers. Outbound calls
now pick grpc.<target>.allowed_server_commonNames (falling back to the
client component's and the global list) while still presenting the
grpc.filer client identity.
* security: scope per-target server allowlists under the client component
The filer read grpc.volume.allowed_server_commonNames when dialing a
volume, but that key already denotes the volume's own outbound allowlist
(e.g. the master identity). A deployment restricting volume->master
calls would make the filer reject real volume servers. Per-target policy
now lives under the client component: grpc.filer.volume_*
falls back to grpc.filer.* then grpc.*.
---------
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* s3tables: answer 404 for a table name that was renamed away
A rename leaves the source directory in place to hold the data under
it, with its catalog attributes cleared. UpdateTable, DeleteTable, the
table policy calls and the table maintenance calls read the metadata
attribute of that directory and reported its absence as an internal
error, so they answered 500, and GetTablePolicy answered NoSuchPolicy.
An absent metadata attribute now reads as a missing entry. tableMetadata
reports it as both filer_pb.ErrNotFound and the attribute error, so
callers that test for either keep working. The policy and maintenance
handlers read the attribute through the new readTableMetadataAttr, which
reports filer_pb.ErrNotFound, so each call answers NoSuchTable.
* s3tables: a view record without its metadata reads absent too
* s3tables: refuse a catalog write that lands on a metadata-less record
---------
Co-authored-by: user.email <chrislusf@users.noreply.github.com>
* helm: add commonLabels to the metadata of every object
The chart lets users label pods (podLabels, <component>.podLabels) and
annotate workloads, but offers no way to label the objects themselves.
Backup, ownership and cost policies usually select StatefulSets,
Deployments, Services, PVCs or Secrets by label, and umbrella charts
that set such labels on all their workloads have to patch this chart
to do it.
commonLabels is added to the metadata labels of every object the chart
renders, through one seaweedfs.commonLabels helper. Values may hold
templates, and the whole value may be a template string rendering to a
map, so a parent chart can pass its own labels down. The chart's
app.kubernetes.io/{name,instance,component,managed-by} and helm.sh/chart
keys are dropped, and keys an object already sets (ingress.labels,
httpRoute.labels, monitoring.additionalLabels, grafana_dashboard) win,
so selectors never change and no key renders twice.
Pod templates are left alone (podLabels covers them, and a label change
would roll the pods), as are volumeClaimTemplates, which cannot change
on an existing StatefulSet. checksum/master-config now hashes the master
ConfigMap as rendered without commonLabels, so labels do not roll the
masters either; its value is unchanged.
With commonLabels empty, the default, every template renders exactly as
before. Adds a helm_ci check that labels a release with most components
on and asserts every object carries the labels and nothing else changes.
* helm: do not reject a commonLabels key literally named Error
fromYaml reports parse failures as an {"Error": msg} map, so checking
for that key also matches a user-supplied label of the same name. Only
treat Error as a parse failure when the raw value was not already a
map, so {Error: tracked} renders while non-map input still fails.
* helm: exclude the filer monitoring label from commonLabels
filer.metricsPort adds a hardcoded monitoring: "true" label, which
collided with a monitoring key in commonLabels and rendered a duplicate
map key. Exclude it while metricsPort is set so the object's label wins,
and keep the user's label when it is not.
* helm: accept a templated commonLabels string with an Error key
fromYaml reports a parse failure as a one-key map {Error: "error ..."}, so a
templated string such as 'Error: x' was taken for a parse error. Only fail
when the value is not a map and fromYaml returned its error shape.
* helm ci: keep ConfigMap data in the commonLabels render comparison
Generated Secrets still need normalizing, but the security ConfigMap was
only non-deterministic because of its default jwt signing key. Disable
that key for this render and compare ConfigMap payloads like everything
else so a commonLabels regression in data is caught.
* helm ci: render the security ConfigMap in the commonLabels comparison
Disabling jwtSigning.volumeWrite alone makes securityConfigEnabled false,
so the ConfigMap was absent from the render instead of compared. Enable
mTLS without the generated key so the object is rendered deterministically
and both its labels and data are covered.
* helm ci: strip only generated-credential secret payloads in the commonLabels comparison
* helm ci: assert the effective non-reserved commonLabels
* helm: render integral numeric commonLabels without exponent
---------
Co-authored-by: jeffWild <20067853+jeffWild@users.noreply.github.com>
Co-authored-by: chrislusf <chrislusf@users.noreply.github.com>
Co-authored-by: Chris Lu <chris.lu@gmail.com>
* deps: bump seaweedfs/raft for PreVote
The default raft gains the pre-vote phase so a master returning from a
partition can no longer depose a healthy leader with an inflated term.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* deps: bump seaweedfs/raft for pre-vote review follow-ups
Picks up the post-merge fixes on top of f227dce: per-round vote workers with per-peer dedupe, pre-vote grants counted at a newer term so mixed-version clusters cannot livelock, and state/conf field locking on the startup paths.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* deps: pin seaweedfs/raft to released v1.2.3
PreVote has merged and shipped as raft v1.2.3, so switch off the
PR-branch pseudo-version.
---------
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* helm: add namespaceOverride to deploy into another namespace
When the chart is a subchart of an umbrella chart, every object lands in
the umbrella's release namespace, with no way to give SeaweedFS its own.
namespaceOverride, empty by default, now sets the namespace of every
object through a new seaweedfs.namespace helper, which falls back to the
release namespace. All uses of .Release.Namespace go through it, so the
in-cluster addresses, certificate SANs, ClusterRoleBinding subjects,
HTTPRoute backendRefs and lookups follow the override too. The
self-signed Issuer, the bucket hook Job and the volume resize hook
objects, which had no metadata.namespace, now set it.
With namespaceOverride unset the default render is unchanged. Adds a
helm_ci check that renders with an override and asserts nothing is left
in the release namespace.
* ci: cover the bucket hook Job in the all-in-one namespace test
The all-in-one render set s3.createBuckets, which bucketHookEnabled
ignores in allInOne mode, so the Job never rendered and the namespace
check could not see it. Use allInOne.s3.createBuckets and assert the
Job is present.
* helm: fail on an invalid namespaceOverride instead of reshaping it
Truncating and trimming could silently map an override onto a different
existing namespace and deploy into it. Validate against the DNS-1123
namespace rules and fail the render instead.
* helm: bound the master data volume name at 63 characters
data-<namespace> can exceed the Kubernetes name limit when the release
or override namespace is near 63 characters; truncate the composed name
and trim a trailing hyphen like the other name helpers.
* helm: quote the rendered namespace in YAML fields
* helm: warn that changing namespaceOverride strands claims and data
* helm: say namespaceOverride only moves namespaced objects
* helm: retrigger the PR head sync (no changes)
* helm: warn that secrets and hook resources do not follow namespaceOverride
* helm: warn that hostPath data does not follow namespaceOverride
---------
Co-authored-by: jeffWild <20067853+jeffWild@users.noreply.github.com>
Co-authored-by: chrislusf <chrislusf@users.noreply.github.com>
* volume.copy: reject only on file-count drift, warn on deleted-count
Go warned and kept the copy on any count mismatch. The file count is
exact on both sides of a stable source, so a mismatch means a genuinely
bad copy and should reject; only the deleted count is a rebuild-time
estimate that legitimately drifts. Match the Rust check_copy_counts,
which warns solely on the counter that can drift.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* volume.copy: stage the copy and validate before dropping the old replica
The destination replica was deleted before the copy even started, so a
copy that failed its checks — now including the file-count validation —
left the server with no replica at all. The copy now writes under a
copying/ staging dir the volume loader never scans, the file-size and
count checks run on the staged .idx (LoadCompactNeedleMap gives the same
counters the mount would), and only then is the existing replica deleted
and the staged files renamed into place. A rejected or failed copy
leaves the previous replica untouched, and the .note marker is no longer
needed since partial staged files can never be loaded.
* volume.copy: remove only the files a failed copy moved into place
* volume.copy: keep the .note in-progress marker through the renames
A crash between the staged-file renames left a loadable volume without
its .vif (writable where the source was read-only) or a partial set the
load scan could not find. Write the .note before the renames and land
the .vif first so any partial state is discovered and swept, then drop
the note before the mount.
* volume.copy: serialize copies of the same volume
Two copies of the same volume write the same staged copying/ file names
and can pass the existing-volume check before the first one mounts.
Fail a concurrent second copy as retryable, matching the Rust volume
server's volume_copies_in_flight.
* volume.copy: restore the replaced replica when the swap fails
Once the staged copy validates, the outgoing replica's files are
hard-linked into copying/ before DeleteVolume unlinks them. If a later
rename or the mount then fails, the preserved files are renamed back
and remounted, so a failed swap no longer leaves the node with no
replica. The links are removed on success.
* volume.copy: clear stale needle-map files around the swap
A leftover .ldb/.sdx/.rdb at the destination base is trusted over the
fresh .idx the mount loads, and a mount that fails leaves the backends
it built. Clear them before the staged files land and again when a
failed copy cleans up, gated on no live replica owning them. Also keep
<base>.vif on cleanup while an .ecx index shares the basename.
* volume.copy: skip the swap when the caller has gone
A caller that left during the copy cannot use the result; bail before
the swap deletes the replica it would replace.
* volume.copy: discard a mounted copy when the result send fails
A mounted copy was already announced to the master, so a later failure
deletes it through the store to withdraw the announcement before the
files go away, rather than unlinking files out from under a registered
replica.
* volume.copy: keep a vif-less copy discoverable on a split index dir
With -dir.idx the staged .idx lands outside the data dir, so a copy
whose source has no .vif leaves nothing the volume scan can find and
the mount fails after the replica it replaced was already deleted. An
empty .vif marks the basename; the loader treats it as absent and
writes a real one on mount.
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* volume.copy: preserve the replaced replica on its own disk
* volume.copy: restore the replaced replica's .vif past a kept incoming one
* test: cover the rollback link/restore helpers
* volume.copy: preserve the mounted replica's own files for swap rollback
The swap backup linked files at the copy's target paths, which are wrong
when the replica sits on another disk or keeps its .idx beside the .dat;
a failed swap then restored nothing. Link the mounted volume's real paths
and require every present file to link before the delete, since a partial
backup cannot roll back. Re-check the store at swap time so a replica
that mounted mid-copy is replaced rather than overwritten, and drop a
.kept incoming .vif that occupies the path the replica's own .vif
restores to.
* volume.copy: clear a stale .idx a mount would prefer over the copy
A load switches dirIdx to the data dir when an .idx sits beside the .dat,
so a leftover <data>.idx shadows the fresh index the copy lands in the
idx dir, and a surviving <idx>.idx fills in when the source had no index
at all. Clearing only the derived backends still let a stale needle map
reach the mount.
* volume.copy: leave the shared copying/ dir for racing copies
Copies of different volumes serialize per volume id but share the disk's
copying/ dir; a finishing copy could rmdir it between another copy's
MkdirAll and its first staged write, failing that copy with ENOENT.
* volume.copy: fail replica preservation on stat errors other than not-found
* volume.copy: refuse a mount while that volume is being copied
A copy claims the volume id, then waits on the source with the
replacement staged. A mount in that window loads the replica whose files
the swap then clears; refuse it with Unavailable, retryable once the
copy settles. Mirrors the same refusal on the Rust volume server.
* volume.copy: refuse configure while that volume is being copied
Configure unmounts and remounts; during a copy that already claimed the
volume id the remount loads files the swap then unlinks. Refuse with
Unavailable, retryable once the copy settles. Mirrors the Rust volume
server's same refusal.
* volume.copy: leave a still-standing replica's index on a rejected copy
* volume.copy: preserve an unmounted replica's files for swap rollback
* volume.copy: claim the in-flight id in mount and configure, not just check it
* volume.copy: keep an EC-shared .vif when displacing an unmounted replica
* volume.copy: restore replaced files over a surviving partial
restoreReplacedVolumeFiles deleted a preserved link whenever the base
name was still taken, on the assumption a landed file there belonged to
the copy. When a moved-file removal fails, that left a surviving partial
paired with the restored set's other files — mixed generations in one
replica.
A leftover at the basename now comes off first so the backup goes back,
and the link is kept whenever it cannot go back.
* volume.copy: link a repeated replaced path once so rollback cannot delete it
* volume.copy: refuse allocation and staged adoption under the copy claim
AllocateVolume and adoptStagedVolume still registered a volume while a
copy of it held the destination: NewVolume loads the index before
registration, and the copy then truncates or replaces files the live
replica already opened, leaving it mounted on a stale index.
Both paths now claim the volume id like VolumeMount and VolumeConfigure,
so registration either wins the claim before the copy starts or sees it
and refuses with Unavailable.
* volume.copy: serialize reload loads with the copy claim
A SIGHUP reload ran loadExistingVolume outside the copy's claim: its
.note branch could unlink a copy's files mid-write, and its
NewVolume+SetVolume could register a replica between the copy's recheck
and its swap, leaving a live volume on a stale index.
loadExistingVolume now consults a per-location hook the server installs:
a vid whose claim is held is skipped for the next pass; otherwise the
load takes the claim and its release drops it after registration, so
whichever side claims the id first owns the files for the operation.
* volume.copy: let a claim holder's own mount past the load guard
VolumeMount, VolumeConfigure, VolumeCopy, and adoptStagedVolume all hold
the per-volume claim while calling store.MountVolume, which reaches
loadExistingVolume where the load guard saw the caller's own claim and
skipped the disk: every mount failed with 'volume not found on disk'.
Plumb claimHeld through MountVolume -> LoadVolume -> loadExistingVolume
via the new MountVolumeClaimed entry point so an owner's own load
proceeds while loads without the claim are still skipped.
* volume.copy: keep the strict count check when only deletes drift
Deletes append tombstone rows without moving the file count, so a copy whose source only deleted mid-flight still has an exactly comparable file count. Gate the strict check on file-count stability alone; the deleted-count drift is already a warning inside checkCopyCounts.
* volume.copy: abort the swap when stale index files cannot be removed
A surviving .ldb/.sdx/.idx is trusted over the copied .idx at mount, so
a failed removal must stop the copy before the staged files land, not be
swallowed. Mirrors the Rust side's remove_partial_files abort.
* volume.copy: clear staged leftovers before starting a copy
The staging names are fixed per volume, so a crashed copy can leave
files under copying/; a retry whose new source lacks an optional file
(.vif, or .dat on a remote volume) would rename the previous run's
bytes into place. Clear them up front and refuse the copy when they
cannot be removed.
---------
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* rust volume: remove the redb index when a failed VolumeCopy cleans up
A copy that mounts and then fails validation is unmounted, not deleted. With
a redb needle map the mount has already built <idx base>.rdb, and
cleanup_failed_copy removed .dat/.idx/.vif/.note but not the .rdb, so the
index database outlived the copy. A later copy of the same volume to that
server reuses it (RedbNeedleMap::try_reuse_rdb keeps an .rdb whose recorded
idx size is not larger than the new .idx and replays only the tail), which is
wrong once the source has been compacted in between.
Move the partial-file removal into CopyDestination::remove_partial_files and
include the .rdb, next to the .idx where Volume::file_name puts it.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Et288CnR1mxDyQNdYNMBNq
* rust volume: clear a stale index before a copy writes, and report cleanup failures
A crashed copy, or one whose partial-file removal failed, leaves a stale
.rdb/.sdx next to the .idx; the next copy's mount would reuse it via
try_reuse_rdb's tail replay. remove_partial_files now also covers .sdx
and returns the first non-NotFound error: callers that can still abort
(the pre-copy clear) do so, while the post-failure cleanup warns so a
surviving index is visible instead of silently poisoning the next copy.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* rust volume: restore the pre-copy leftover clear
A leftover index check-in accidentally gated remove_partial_files behind
a constant false, so a stale .rdb/.sdx at the destination survived the
copy that was meant to clear it.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(volume): expect 409 when a read-only upload is refused
x-seaweed-write-outcome: refused-before-append. The HTTP integration
test still expected 500, so Rust unit tests failed on Linux and Windows.
Deletes stay 500.
* rust volume: keep a shared .vif, and clear a stale index beside the .dat
remove_partial_files deleted <data>.vif even when a sealed .ecx still
shares it, undoing the keep in Volume::destroy. It also cleared the
index only under the index base, so a leftover .idx beside the .dat
still made load trust that directory's .rdb.
Skip .vif while a non-empty .ecx is present, and clear both bases when
the index directory differs from the data directory.
* rust volume: do not delete .dat files in the index directory
The index directory can be another location's data directory. Clearing
.dat and .note there removes that replica. Those files come off the
data base only; .idx, .rdb, and .sdx are still cleared on both.
* rust volume: serialize copies of a volume on one server
A second VolumeCopy of the same volume could pass the existing-volume
snapshot before the first copy mounts, then clear or overwrite the
files the first copy still owns. Hold a per-volume claim for the whole
copy, like ec_decodes_in_flight does for decodes, and fail a duplicate
fast so the caller can retry. Mirrors the Go volumeCopiesInFlight
guard.
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* rust volume: keep files when the mounted-copy delete is refused
When a copy mounted and then failed, deleting through the store can be
refused (e.g. the volume is compacting), leaving it registered. The
cleanup then unlinked its files anyway, stranding a live replica. Keep
the files while the volume is still in the store. Mirrors the Go copy
cleanup.
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* rust volume: init volume_copies_in_flight in the http test state
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* rust volume: refuse a mount while that volume is being copied
volume_copy claims the vid, snapshots the store, then waits on the
source. A VolumeMount in that window can load the replica, and the
copy's pre-write cleanup then unlinks its files. Check the claim under
the store write lock and return Unavailable.
* rust volume: refuse configure while that volume is being copied
volume_configure unmounts and remounts. A copy that already snapshotted
the volume as absent can then unlink the files that remount loaded.
Check the copy claim under the store write lock and return Unavailable.
* rust volume: preserve files a copy displaces so a failed copy restores them
* rust volume: keep an interrupted copy's .note with the files it marks
displace_existing_files skipped the .note, so a crashed copy's leftovers
were saved without the marker the load scan rejects them by, and
remove_partial_files then deleted it. A retry that failed restored the
partial set marker-free: after a restart the server could load an
unfinished copy as a normal replica and serve wrong data.
Displace the .note like every other file at the basename so the restore
puts back the exact state the copy found.
* rust volume: restore displaced files over a surviving partial
restore_displaced_files dropped the backup whenever the basename was
still occupied. When the partial cleanup could not remove every file the
copy landed, an existing path just meant the removal failed — deleting
the saved file destroyed the old replica and let the surviving partial
pair with restored siblings of another generation.
Clear the leftover first so the displaced file goes back, and keep the
backup when it cannot: the operator can recover from copying/ rather
than lose the replica outright.
* rust volume: back up an EC-shared .vif the copy truncates in place
A .vif a sealed .ecx shares was left out of the displaced set so the EC
volume keeps it — but the incoming copy opens the same path with
truncate, so a failed copy left the shared .vif overwritten by the
rejected copy's settings with no way back. Go keeps the original via a
hard link because its staged file swaps in by rename; truncating the
shared inode makes a link useless here.
Copy the content into copying/replaced_ like the other backups instead:
the file stays in place for the EC volume during the copy, and the
restore puts the original settings back on failure.
* rust volume: serialize reload loads with the copy claim
A SIGHUP reload scans every volume basename and sweeps .note-marked files
as interrupted copies — including one a live VolumeCopy is still writing.
Mirror Go's loadGuard: the copy-claim set is shared with each disk
location, the scan skips a claimed vid outright and claims any other
through its load, and reconcile_compact_states takes the same skip before
rolling a leftover commit over the copy's files.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* rust volume: hold the load claim through open_volumes
The scan released each claim when its iteration ended, before the queued
volume's index read ran in open_volumes. A VolumeCopy taking the vid in
that gap could displace or remove files out from under Volume::new — the
register the guard exists to prevent. Queued claims now live until the
open pass returns; a second collection of an already-claimed vid merges
into the entry instead of re-claiming.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* rust volume: refuse allocation while that volume is being copied
AllocateVolume registered a volume while a copy of it held the
destination: create_volume loads the index before registration, and the
copy then clears or rewrites files the live replica already opened,
leaving it mounted on a stale index — the window VolumeMount and
VolumeConfigure already refuse.
Claim the volume id like the Go AllocateVolume does, so registration
either wins the claim before the copy starts or sees it and refuses
with Unavailable.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* rust volume: refuse staged adoption while that volume is being copied
The from_staged branch of volume_ec_shards_to_volume renames .copying
files into place and mounts, all while a copy of the same vid could be
rewriting the destination — the same registration window VolumeMount
refuses. Hold the copy claim for the whole adoption, as Go's
adoptStagedVolume does.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* rust volume: drop a partial .note when its write aborts the copy
A write_note failure after displace_existing_files restores the saved replica but leaves the empty or partial marker it just made; the next load scan then deletes the restored replica as an unfinished copy. Remove the marker before restoring, as the Go side's error defer does.
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: user.email <chrislusf@users.noreply.github.com>
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: chrislusf <chris.lu@gmail.com>
* rust volume: keep a copied replica on deleted-count drift, as Go does
check_copy_counts rejected a VolumeCopy whenever the target's deleted count
differed from the source's. On a redb needle map that count is an estimate:
metrics_from_idx rebuilds it at load through a bloom filter (0.1% false
positives per entry, each counted as an overwrite), and the source holds the
estimate from its own last load plus exact runtime updates. A source written
since its load and a fresh load of the same files disagree by a few, so good
copies were refused with "target deleted count [N+1] is not same as origin
deleted count [N]". An index of unique keys 1..=128 already reports one
deletion.
Go fixed the same rejection in #11701 (issue #11678) by warning instead of
failing. Do the same for the deleted count. The file count stays strict: it
is exact on both sides (every non-tombstone .idx row is one file), and the
.dat/.idx size checks still gate integrity.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Et288CnR1mxDyQNdYNMBNq
* test(volume): expect 409 when a read-only upload is refused
x-seaweed-write-outcome: refused-before-append. The HTTP integration
test still expected 500, so Rust unit tests failed on Linux and Windows.
Deletes stay 500.
* rust volume: keep the strict count check when only deletes drift
Deletes append tombstone rows without moving the file count, so a copy whose source only deleted mid-flight still has an exactly comparable file count. Gate the strict check on file-count stability alone; the deleted-count drift is already a warning inside check_copy_counts.
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: user.email <chrislusf@users.noreply.github.com>
* volume.copy: drop the needle-map index a failed copy leaves behind
A VolumeCopy that mounts and then fails validation is only unloaded, not
deleted, so a leveldb (.ldb) or sorted-file (.sdx) index built by the
mount outlives the copy. The next copy of the same volume to this server
then trusts the stale index (generateLevelDbFile replays only past its
watermark), which is wrong once the source has compacted in between.
Mirrors the Rust server's remove_partial_files.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* volume.copy: clear a stale index before a copy writes
A crashed copy, or one whose cleanup failed, leaves a stale .ldb/.sdx/.rdb
next to the .idx that this copy's mount would trust (generateLevelDbFile
replays only past the stored watermark, and a same-size .idx keeps every
un-replayed offset). Abort on a removal error rather than risk a mount
reusing a poisoned index. Mirrors the Rust server's pre-write
remove_partial_files call.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* volume.copy: keep a mounted replica when the final response send fails
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* volume.copy: discard a mounted replica the caller abandoned
A send failure after MountVolume means the caller is gone, but the
replica was already announced to the master. Deleting its files under
the live mount breaks the loaded index; leaving it mounted keeps an
orphan the master routes to. Mirror the Rust copy cleanup: track the
mount, and on error delete through the store (unmount + withdraw the
announcement + remove files, keeping remote data), and check the stream
context before mounting.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* volume.copy: serialize copies of a volume on one server
A second VolumeCopy of the same volume could pass the existing-volume
check before the first copy mounts, then clear or overwrite the files
the first copy still owns. Mirror ecDecodesInFlight: the claim covers
the whole copy, and a duplicate fails fast so the caller can retry.
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* volume.copy: remove only files the copy owns, and every stale index
Three cleanup gaps around which files a copy may remove:
- The failure cleanup ran on any error, so an abort before the first
write (a failed stale-index clear or a failed .note write) could
delete a previous replica's files the copy had not touched. File
removal is now gated on the copy having started writing.
- A mount that finds an .idx beside the .dat builds the needle-map
index under the data base, so leftovers there poison the next copy
the same way the idx-dir ones do. Clear and clean up .idx/.ldb/
.sdx/.rdb under both bases when the dirs differ.
- A regular volume and an EC volume for the same id share <base>.vif.
Keep it while the EC index is on the disk, as the destroy path does.
- When the discard of a mounted copy is refused (e.g. it is compacting)
the volume stays registered; unlinking its files would strand a live
replica.
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* volume.copy: do not unlink a previous replica on an early abort
An abort before the first write now leaves the destination files alone,
but three removal paths still leaked past that gate: a partially-written
.note, and the post-copy defer's .idx/.vif at the data base. Remove the
.note unconditionally (it only ever marks an in-flight copy) and gate the
data-file removals in the outer defer on copyOwnsFiles too.
* volume.copy: refuse a mount or configure while a copy is in flight
A copy claims the volume id, then waits on the source before it clears or
overwrites the destination files. A mount or configure landing in that
window loads the replica whose index files the copy is about to remove.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* volume.copy: abort when a replica registers during the source RPCs
The existing-volume snapshot runs before the copy calls the source, so a
mount that lands in between escapes it. Recheck registration just before
the destructive clear, which can share the index directory with the
mounted replica's bases.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* volume.copy: keep files whenever a replica of the volume stays registered
The cleanup already kept files after a refused delete; a replica a racing
mount registered during the copy needs the same protection, since its
index directory can coincide with the bases being cleaned.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* volume.copy: skip the index cleanup when the copy never picked bases
An early failure leaves the base names empty, so the removals would
otherwise run against ".ldb"/".sdx"/".rdb" in the process directory.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* volume.copy: serialize mount and configure registration under a per-volume guard
The in-flight flag check in VolumeMount and VolumeConfigure was not
atomic with volume registration, and the copy's registration recheck was
not atomic with its stale-index clearing. A mount that passed the check
just before the copy claimed the volume could register after the recheck
and then lose its index files to the clearing.
Replace the bare flag with a per-volume guard: the copy claims it for
the copy's life, mounts and configures hold it across check plus
registration, and the copy holds it across the recheck plus clearing.
Whichever side takes the guard first wins, so a mount either registers
before the recheck (and aborts the copy) or sees the claim and refuses.
Guards persist in the map so a mutex handed to a waiter is never
replaced mid-wait.
* volume.copy: refuse allocation and staged adoption under the copy guard
AllocateVolume and adoptStagedVolume still registered a volume while a
copy of it held the destination: NewVolume loads the index before
registration, and the copy then truncates or replaces files the live
replica already opened, leaving it mounted on a stale index.
Both paths now take the per-volume guard through their registration, so
they either register before the copy's recheck (and make it abort) or
see the claim and refuse with Unavailable.
* volume.copy: serialize reload loads with the per-volume guard
A SIGHUP reload ran loadExistingVolume outside the copy's guard: its
.note branch could unlink a copy's files mid-write, and its
NewVolume+SetVolume could register a replica between the copy's recheck
and its file clearing, leaving a live volume on a stale index.
loadExistingVolume now consults a per-location hook the server installs:
a vid with a copy in flight (or whose guard the copy currently holds) is
skipped for the next pass; otherwise the returned release keeps the
guard held through registration, so the copy's recheck either sees the
volume and aborts or clears first.
* volume.copy: skip compaction-state reconcile while a copy owns the volume
* volume.copy: let a guarded operation's own mount past the load guard
VolumeMount, VolumeConfigure, adoptStagedVolume hold the volume guard
locked, and VolumeCopy holds copying=true, while calling store.MountVolume.
The mount reaches loadExistingVolume where the new load guard saw the
caller's own hold (TryLock fails, or copying set) and skipped the disk:
every guarded mount failed 'volume not found on disk' — a completed copy
deleted its copied files, configure left the volume unmounted, staged
adoption failed after renaming.
Plumb ownLoad through MountVolume -> LoadVolume -> loadExistingVolume via
the new MountVolumeClaimed entry point so the owner's own load proceeds
while unrelated loads are still skipped.
* volume.copy: fix own-load test call, assert unguarded mounts still skip
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* volume.copy: refuse unmount and delete while a copy owns the volume
---------
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* filer.backup: add -s3FlattenVersions to mirror the current S3 view
Versioned buckets are stored as <key>.versions/v_<versionId> files plus a
latest-version pointer on the .versions directory, so a backup mirror held
internal version files and could never be restored through the S3 API.
With -s3FlattenVersions the backup maps events on .versions directories
onto the flattened object key: a new latest version uploads to <key>, a
latest delete marker or a cleared pointer deletes <key> when
-doDeleteFiles is set, and the suspended-versioning null object resolves
to the regular path. Version files themselves are never mirrored. The
initial snapshot seeds only the version the pointer names.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* filer.backup: serialize flattened writes per object key
Same-path non-barrier dir updates run concurrently in the metadata
processor, so two pointer updates for one object -- or a pointer update
and a regular-path write to it -- could overlap and let the older upload
finish last. Lock on the flattened key: versions events lock the object
key, every other event its own path, via a striped mutex.
* filer.backup: apply exclusions to the flattened object key
File-name and path excludes ran against the .versions entry, so an
excluded object was still copied flat and its deletion still removed the
destination copy. Match the rules against the flattened key on both the
apply and the versions-dir-removal paths.
* filer.backup: re-resolve the pointer on version-file events
A tag or attribute update on the current version touches the v_* file,
not the .versions pointer, so dropping every version-file event left the
mirror with stale tags until the next pointer flip. Re-resolve the
pointer on each version-file event; noncurrent versions still stay out.
* filer.backup: tolerate vanished versions dirs in flattened snapshot
The flattened seed path returned source errors before the existing
missing-source policy ran, so one version deleted mid-walk could force a
full re-walk. Apply the same 404/superseded skip as the generic seed
branch, and cover the review fixes with tests.
* filer.backup: lock both flattened keys an event touches
A rename or delete can retire one flattened key while writing another,
so serialize on the old- and new-side keys together.
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* filer.backup: refresh the flattened key only for the still-current child
A v_* event on a non-pointed version cannot move the flattened key, so
skip the re-upload; tag and attribute updates on the current version
still flow through.
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* replication: let resolved-latest writes bypass the source staleness check
A filer sink skips a CreateEntry whose source path no longer holds the
entry, but a flattened write lands on a key whose source path is not the
one it was read from, so the check dropped every flattened upload.
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* filer.backup: reclaim destination chunks on flattened deletes
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* filer.backup: route regular-path events for versioned keys through the pointer
A null-object write or delete at the regular path can coexist with a
.versions pointer that names a live version; the pointer decides the
flattened key, so deleting the null object must not remove it.
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* filer.backup: make the checkpoint identity cover the flattening mode
Resuming a checkpoint taken in the other mode would skip the flattened
re-seed and leave mirrored version files behind; a mode change now
starts a fresh snapshot.
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test: cover flattened regular-path routing and per-mode checkpoints
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* filer.backup: resolve both sides of a flattened rename
* test: cover flattened renames with one versioned side
* replication: let resolved-latest writes overwrite a newer destination
The stale-destination skip trusted a newer destination mtime, but a
resolved-latest write can legitimately install an older version over the
copy of a since-deleted latest. Skip only on provably identical content
(matching chunk provenance, or equal ETag with full byte coverage).
* filer.backup: rescan .versions when the latest null object is gone
Deleting ?versionId=null removes the regular-path object without touching
the null-is-latest pointer, so the flattened key kept its stale copy.
Fall back to the newest remaining version (or a flat delete), the same
rescan the S3 read path applies.
* replication: compare inline and remote content in the destination-holds checks
A chunk-less entry keeps its bytes in Content and a remote-only entry in
the object RemoteEntry points at, but the identical-content shortcuts
only compared chunk lists and chunk-derived ETags. Two different inline
files looked identical, so a flattened re-copy or a same-mtime rewrite
kept the previous version's bytes. Compare Content bytes and remote
object identity wherever the checks claim the destination already holds
the reference's content.
* filer.backup: paginate the null-latest .versions rescan
A limit=0 listing is capped at DirListingLimit server-side, and
old-format version names sort oldest-first, so a history longer than one
page left the flattened copy on an older version after the null object
was removed. Page through the directory like the S3 read path does.
* replication: require matching writable metadata for resolved-latest skips
A skip keeps the destination's stored metadata, so identical bytes were not
enough: stale attributes or extended entries (e.g. a superseded version id)
survived under equal content. Compare the fields a rewrite controls —
attributes minus filer-assigned inode/crtime, with the store's atime/ctime
zero-defaults applied — plus the extended map, in both the provenance and
ETag fast paths.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* replication: require the extended key to exist when comparing metadata
* replication: ignore a stored rule TTL when the resolved write carries none
* filer.backup: rename -s3FlattenVersions to -s3.flattenVersions
Match the -s3.* flag naming convention used by the other s3 options.
---------
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* s3sink: upload through a concurrent-safe ReaderAt so parts do not thrash
s3manager splits a seekable body into parallel section readers, and every
part's ReadAt landed on the single chunk buffer in ChunkStreamReader: parts
at different offsets kept evicting each other's chunk and refetching it
whole, which produced massive read amplification against the volume
servers and collapsed throughput.
Build the upload body from filer.ChunkReadAt instead, the same
concurrent-safe reader the mount and query paths use: each part's range is
served from a shared reader cache with parallel chunk fetches and
range-reads for clipped views, so every byte is fetched once per pass.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* filer: buffer recently fetched chunks for concurrent readers
* Revert "s3sink: upload through a concurrent-safe ReaderAt so parts do not thrash"
This reverts commit be1ec2d7c7.
---------
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* seaweed-volume: pin the read-only write status to 409 in the test
PR #12236 moved read-only upload refusals from 500 to CONFLICT with the
refused-before-append proof; the test still asserted Go's 500 for the
write. Deletes on a read-only volume keep surfacing 500.
* seaweed-volume: correct the read-only status comment in the test
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* s3tables: refuse a create that a namespace delete can lose
DeleteNamespace checks the namespace for children and then deletes its
directory recursively in a second filer call. The filer locks each path
on its own, so a table created under the namespace between the two is
not seen by the check and is removed by the delete, while both requests
answer 200. The mirror case loses the table too: a create that passed
its namespace check before the delete finished re-created the namespace
directory through the filer's parent creation and recorded a table in a
namespace that no longer exists.
Each side now writes before it reads. DeleteNamespace claims the
namespace record with a fresh token before it scans, deletes only while
the record still holds that claim, and lifts its own claim when it
stops. CreateTable, RegisterTable and CreateView re-read the namespace
after writing their record and back out when it is claimed or gone:
they remove what they wrote and answer 409 or 404 instead of reporting
an entry that does not survive, or 500 when the record could not be
removed and may remain. An entry the scan missed was written after the
claim, so its create sees the claim; a create and a delete can both
fail, but no longer both succeed. A create is also refused up front
while a claim is held, but only after its authorization check, so a
caller that may not create there does not learn of the delete.
A create writes its record marked "create" and lifts the mark once the
namespace is confirmed open. Until then GetTable, GetView, UpdateTable
and a second create treat the record as absent, so no request is told
about a table or view that the first create may still back out.
* s3tables: keep a create's unfinished record out of reads and listings
* s3tables: refuse catalog writes on a record an operation claimed
* s3tables: answer 404 when a create's write path loses the namespace
* s3tables: confirm the rename destination's namespace like a create
* s3tables: cover unfinished-record reads, claimed-record writes, rename claim
* s3tables: lift a leftover claim when a no-token delete clears the record
* s3tables: answer 409 when a claimed record turns away a write
UpdateView, TagResource, and UntagResource surfaced ErrConcurrentUpdate
as a 500, so an expected conflict looked like a server fault to the
caller. Map it to Conflict the way UpdateTable and the policy handlers
already do.
* s3tables: name the lost source when a rename finds it deleted
* s3tables: report an entry deleted before its update as not found
The filer answers an UpdateEntry on an entry removed since the lookup
with codes.NotFound. mutateEntryExtended returned that error unchanged
instead of filer_pb.ErrNotFound, so a rename whose source was deleted
mid-claim answered 500 instead of NoSuchTable or NoSuchView.
MemFiler now answers a missing entry the same way, so the rename tests
for tables and views take the real filer's path.
* s3tables: unpublish the destination when a rename's source clear fails
* s3tables: refuse to delete a record another operation claimed
* s3tables: release catalog claims on an uncanceled cleanup context
* s3tables: answer 404 when a racing delete already removed the table
* s3tables: keep the rename destination marked until the source clears
* s3tables: guard the rename rollback against a recreated source and an unfinished destination
* s3tables: drop the unfinished destination only once the source is restored
* s3tables: re-resolve the purge target on the record a delete claims
* s3tables: cover a delete that meets a moved metadata location
* s3tables: answer 409 when a create's record changed under it
* s3tables: adopt a record still marked by its unfinished create
* s3tables: skip unfinished creates in the namespace empty check
* s3tables: delete only the record the caller was authorized on
DeleteView and DeleteTable check permissions on the record they read
first, then read the name again to claim it. A record that took the name
in between, such as a table or another owner's view created after a
concurrent delete, was claimed and removed on the first record's
authorization, and DeleteView never compared the caller's version token
with it.
The second read now repeats the first one's checks: a record of the
other kind or an unfinished create answers 404, and a token that no
longer matches answers 409. A delete without a token also compares the
record's version token with the authorized one and answers 409 when they
differ.
* s3tables: re-check the parent after a create lifts its mark
A create that passed checkParentOpen and paused before lifting its mark
could still confirm while a namespace delete held the claim: the delete
scans the still-marked record, skips it, and removes it under a create
that already answered success. confirmParent now re-reads the same
parent record after the mark clears and backs the confirmed record out
when the namespace moved on.
* s3tables: mark the rename destination with a claim of its own
claimCatalogEntry adopted every create-marked record, including a
rename's destination: a create landing after the source cleared could
replace the destination before the rename published it, and with the
source name reused the restore could leave the table on neither name.
The destination now marks with the rename's own claim, which readers
still treat as unpublished but a create must refuse, and the commit
re-checks the destination namespace after the record goes live. The
source keeps its rename claim until the commit lands so a namespace
delete scanning the cleared record still refuses.
* s3tables: compare the parent's createdAt, not its editable metadata
A namespace update rewrites ExtendedKeyMetadata in place, so a create or
rename whose second parent read landed across an UpdateNamespace saw a
changed blob and reported the still-live namespace as gone: creates backed
their record out and answered 404, renames undid their destination.
createdAt survives the edit but changes on a delete+recreate, so it still
identifies the record.
* s3tables: retry lifting the rename's claim off the cleared source
The release's only failure mode is a transient filer error — nothing else
can write the cleared record — yet a single loss left the claim behind:
the old name stayed refused to creates and the namespace counted the
record as a child. Retry the release before warning.
* s3tables: page the namespace scan before deleting it
A page of skippable entries — bare directories or unfinished creates —
ended the scan with hasChildren false, so a table on a later page was
removed with the namespace. Keep listing with StartFromFileName until a
child shows or the listing ends.
---------
Co-authored-by: chrislusf <chrislusf@users.noreply.github.com>
* test: add s3 integration tests for SQLite Cloud Backed Storage (#12235)
SQLite CBS stores a database as named objects in a bucket and relies on
conditional PUT — put-if-absent for new blocks and ETag compare-and-swap
on the manifest — so concurrent clients cannot corrupt the database.
The suite runs a weed mini cluster (conditions are evaluated under the
object's write lock on the owner filer via the routed-write path) and
covers:
- the CBS storage primitives: create/put/fetch/list/delete/destroy
- If-None-Match: * put-if-absent, success and 412 rejection
- If-Match ETag compare-and-swap, including stale-ETag rejection and
the re-read/retry loop CBS clients run on 412, on plain and
versioned buckets
- racing put-if-absent and manifest CAS writers: exactly one commit
wins, every update survives
- conditional CompleteMultipartUpload, sequential and raced
- an end-to-end bcv-style upload, download-and-verify, and checkpoint
A matching job is added to the S3 mutation regression workflow.
* test/s3: drop issue references from the sqlite cbs suite comments
* test/s3: abort every losing upload after the conditional multipart race
* test/s3: assert the winning writer's body after the conditional write races
* test/s3: spell out the get-returned etag in the cbs suite
* test/s3: assert the completed multipart ETag the cbs suite feeds back
The conditional-completion checks pass completeOut.ETag into If-Match;
if the response carried no ETag the check would silently run
unconditional.
* test/s3: drop the unused generation parameter from the cbs block body
* test/s3vectors: data-plane lifecycle integration test
TestVectorDataPlaneLifecycle spawns a real weed-vector against the lance
listener, mirrors the pytest cluster's flags, waits for the gateway to
discover the registered engine, and walks the full lifecycle: create
bucket and index, put/get/list vectors, a filtered query that excludes the
nearer non-matching vector, delete vectors, index, and bucket. The test
skips when no engine binary resolves (VECTOR_BINARY, ./weed-vector, PATH).
* test/s3vectors: fail fast when the engine dies before listening
* test/s3vectors: harden engine launch and tighten lifecycle assertions
* test/s3vectors: wait on the engine's exit once through a broadcast channel
* s3vectors: record request metrics on the dispatch path
The S3 API tracks every operation through S3RequestHistogram,
S3RequestCounter, and S3InFlightRequestsGauge; the vectors dispatch rode
the same listener with no instrumentation. The op name labels each
series under s3vectors.<Operation>.
* s3vectors: count auth rejects and keep bucketless metrics alive
* s3vectors: bound GetVectors and DeleteVectors key lists
Both forwarded req.Keys to the engine unbounded; AWS caps these at 500,
the same window a PutVectors batch shares.
* s3vectors: cap GetVectors at the AWS 100-key limit
GetVectors allows at most 100 keys per request while DeleteVectors
allows 500, so give vectorKeysBounds the per-call maximum instead of
sharing the put-batch window.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* s3vectors: assert out-of-window key lists never reach the engine
Record get/delete calls on the fake engine and cover both handlers
rejecting empty and oversized lists with 400 ValidationException.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
filer.data, filer.logs and master.logs accept type existingClaim and render
the pod volume for it, but the volumeMount (and, for logs, -logdir) only
covered hostPath, persistentVolumeClaim and emptyDir. With
filer.data.type=existingClaim the claim was attached to the pod but never
mounted, so the filer kept its metadata store on the container's own
filesystem and lost it on every restart. admin and worker already include
existingClaim in the same conditions.
Adds a helm_ci check that renders both StatefulSets with existingClaim and
asserts the mounts and -logdir.
Co-authored-by: Pat White <5015863+patwhite@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* webdav: answer 204 to a PUT that replaces a file
* webdav: decide the 204 from what OpenFile actually replaced
A Stat before the write left a window where a delete or create in
between made the status wrong, and it cost a second filer lookup. The
OpenFile create path already knows whether it removed an existing
entry, so it now marks the request context and the writer maps 201 to
204 only on a real replace.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test/webdav: cover OpenFile's own put-outcome flag for both create paths
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* webdav: take the put outcome from the create's OldEntry
A stat-then-create still races: a second writer landing between the
check and the write makes the reported status describe a state the
write never saw. The filer decides create-vs-replace under its per-path
lock and reports the previous entry in the response's metadata event,
so flag the PUT replace from OldEntry instead of a pre-write stat. The
in-place update keeps the replaced entry's crtime and inode and reuses
the filer's own chunk cleanup, so the separate removeAll goes away, and
an exclusive create now defers to the filer's atomic O_EXCL instead of
a racy local check.
* webdav: send no body with the 204 a replacing PUT answers
---------
Co-authored-by: user.email <chrislusf@users.noreply.github.com>
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* volume server: add idx_content to VolumeIncrementalCopyResponse
A remote-tier delete is an index-only tombstone row with no .dat record,
so it cannot reach an incremental copy through the .dat stream. Carrying
the .idx tail verbatim lets the backup reproduce those rows at their
exact positions.
* backup: carry index-only tombstones through incremental copies
A remote-tier delete leaves an offset-0 tombstone in .idx with no .dat
record. The copy only streamed .dat bytes and regenerated .idx from
them, so the tombstone never reached the backup: the deleted needle
stayed live and every later incremental diverged and rebuilt again.
backupCursor and ResumeByLastEntry now accept an index-only tombstone
as a boundary, matched by key and size since there is no record to read
a cookie from. VolumeIncrementalCopy appends the .idx tail after the
.dat tail, and IncrementalBackup writes it at the index boundary,
truncates, and loads the new rows — so tombstones land at the exact
positions the replica has them.
* seaweed-volume: mirror index-only tombstone handling in incremental copy
resume_by_last_entry accepts an offset-0 tombstone boundary matched on
key and size, skips its missing cookie check, and resumes .dat streaming
at the next entry that has a record — index-only rows ride the new
idx_content stream instead of reporting divergence. The server captures
the .idx bound before the .dat one (a row is published only after its
bytes) and streams the tail after the data.
* backup: keep incremental copy bounds and streams consistent
Review fixes:
* Bound capture moved after the resume reads and reads .dat/.idx sizes
together (Go FileStat, Rust file_stat under one lock), so every row
resume saw is inside the index bound and each row's record is always
inside the data bound.
* A compaction commit changes CompactionRevision: the server now rejects
the copy when the revision moved between resume, file opens, or the
end of the stream, so a backup never mixes revisions. The .idx is
opened up front to pin the inode.
* An index tail that ends before the captured bound is an error, not a
silent success.
* IncrementalBackup rolls the .idx tail and .dat bytes back to the
pre-copy boundary when the stream fails, so a partial row cannot wedge
backupCursor and unapplied rows cannot pass for current ones.
* The index tail is replayed through a small indexTailReceiver interface,
covering the leveldb needle maps too, not just the in-memory map.
* backup: bound the resume scan to the captured index size
The resume and the stream bounds are now captured in one order:
revision, then the .dat/.idx pair, then the scan. ResumeByLastEntry and
resume_by_last_entry take the index bound and scan no further, so a row
appended while the copy is being set up can never enter the index
stream without its .dat bytes.
The index-tail replay stays on the in-memory map only: the leveldb
map's DoOffsetLoading writes to the .cpldb compaction database, not the
live one, so it cannot serve a live tail replay; other maps keep
regenerating the index from .dat as before.
* seaweed-volume: guard incremental copy total against a located offset past .dat
A corrupt index can place the resume offset beyond the captured .dat
size; has_data already refuses the reader, but total still evaluates
dat_size - s and underflows.
* backup: bound the incremental .dat tail by index-covered records
A record staged but not yet indexed can sit inside the captured .dat
bound while its row stays outside the index bound — the fsync gap
between staging and publishing is not covered by one stat. Streaming
those bytes leaves them unindexed on the backup, and the next run
appends the record again. Bound the .dat tail by the last record the
streamed rows cover on both producers, and cover the bound walk with
tests.
* backup: cap the .dat stream at the captured bound and roll back replay failures
sendFileContent could read past stopOffset on its last chunk, so a
needle appended after FileStat arrived without its index row and the
next copy appended at the wrong position. Each read is now limited to
the remaining captured bytes.
A failed truncateIndex or DoOffsetLoading after a successful stream
left the index tail on disk unapplied, which a later backupCursor could
mistake for applied rows; both now roll the tail back to the pre-copy
boundary like the stream-error path.
* backup: bound the incremental .dat tail by the maximum covered offset
A key-sorted .idx ends on the highest-key row, which need not be the
record physically last in .dat, so the last-row bound could cut off
live records. Take the end of the highest-offset row inside the bound
instead, and cover the case with a sorted-idx test in Go and Rust.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* backup: restore .dat and the needle map when index replay fails
A failed DoOffsetLoading left the received .dat bytes in place and could
leave partial mappings for rows the index rollback just removed, so the
next copy appended the same records again or read stale entries. Truncate
.dat back to the copy boundary and restore the touched mappings and
metrics from a pre-replay snapshot.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* backup: cover no index rows with the superblock bound
When no row inside the index bound names a record — an empty index or a
bound of index-only tombstones — the covered end is the superblock, not
the captured .dat size, so staged bytes are never streamed without a
covering row.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* seaweed-volume: apply the covered bound even when the index streams no rows
The bound was gated on idx_end > idx_start, so an empty captured index
skipped it entirely and a staged .dat tail still shipped. Gate on the
stream-index predicate instead, matching Go's streamIndex check.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* backup: propagate index coverage read errors
* backup: rebuild the needle map from the truncated index on replay failure
* seaweed-volume: propagate index coverage read errors
---------
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* webdav: overwrite on MOVE when the Overwrite header is missing
* webdav: defer a MOVE destination's delete into the rename
x/net removes an existing destination before Rename, so a refused or
failed move — a CanRename rejection, a vanished parent, a transient
filer error — loses the target entirely. For an overwriting MOVE onto
an existing destination, run the request against a filesystem that
skips the destination RemoveAll and lets the filer's rename replace the
target inside one transaction instead; a file vs. collection clash
still takes the delete + rename path. Rename itself now tolerates an
existing target, which the atomic filer rename already handles.
* webdav: only default Overwrite when the header has no values
An Overwrite header sent with an empty first value makes Header.Get
return "", which the defaulting treated as absent and rewrote to T —
overriding a client's explicit F in a later value. Check the value list
instead so a present-but-empty header is left alone.
* webdav: move a collection onto a non-empty one via delete + rename
deferredDeleteFS skips the destination RemoveAll so the filer rename can
replace the target in one transaction, but the filer only replaces an
empty collection: moveSelfEntry deletes the target non-recursively and
refuses ErrNonEmptyFolder. Detect a non-empty collection destination and
take the depth-infinity delete + rename path moveFiles would have run.
* webdav: park a clashing MOVE target aside until the rename lands
A refused rename (cross-collection move, bucket target) after deleting
the destination lost its contents; the target now renames to a hidden
sibling first and is restored when the winning rename fails.
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* webdav: restore a parked MOVE target under a detached context
A client disconnect cancels the request context mid-move, which also
canceled the restore and stranded the destination under its hidden
sibling name; the restore and parked-entry removal now run under
context.WithoutCancel with a bounded deadline.
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* webdav: bound the parked MOVE target's restore after the move
The detached restore deadline started before the source rename, so a
long move could exhaust it; each detached context now starts where its
work begins. A failed parked-entry removal retries briefly and its
warning names the parked sibling for recovery.
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* webdav: restore a parked MOVE target without replacing a new claimant
A client can create the destination name while the old target is parked, so the restore rename must fail instead of replacing that new entry.
* webdav: propagate RemoveAll's context to the filer delete
The parked-target cleanup runs under a bounded detached context, but the delete RPC discarded it; passing ctx through makes the deadline effective.
---------
Co-authored-by: user.email <chrislusf@users.noreply.github.com>
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test: move the s3vectors pytest suite under test/s3vectors
The vectors API tests lived in test/vectors next to the new s3vectors
integration home; the rust-vector-tests workflow's paths and working
directory move with it.
* gitignore: cover the binaries the s3vectors Makefile builds
* test/s3vectors: integration test design and catalog-surface suite
The design doc lays out a two-tier suite: catalog/policy ops that need no
vector engine run against a real weed server over HTTP, while the data
plane skips itself when no engine is reachable. The first test brings up
the server and exercises the bucket/policy lifecycle, encryption-config
round-trip, and pre-engine validation boundaries.
* test/s3vectors: sign API calls so requests reach the vectors router
The gateway selects the vectors router off the SigV4 credential scope —
unsigned requests with only X-Amz-Target fall through to the plain S3
router, so every call in the suite failed before dispatch. Sign each
request with the s3vectors service name and point the spawned server at
a written identity file so the signature is actually verified.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test/s3vectors: bound the spawned server's ports, listeners, and clients
weed server derives each gRPC listener as HTTP+10000, so ephemeral
allocation could hand back a port whose pair is invalid or taken; the
-s3 defaults also bind the Iceberg and Lance listeners. Allocate the
HTTP/gRPC pairs together, pin -ip/-ip.bind, disable the extra listeners,
and give the readiness probe and API client deadlines.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test/s3vectors: verify policy round-trip and gate the engine skip on 503
The stored policy check accepted any JSON body, and a delete that left
the document behind could not be caught — compare the returned document
verbatim and assert a follow-up Get returns 404. The engine probe never
created its bucket, so a healthy engine still produced a skip; create it
first, skip only on a confirmed 503, and fail on any other error, then
exercise the catalog-side index ops when an engine answers.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
s3: AbortMultipartUpload answers NoSuchUpload for an upload that no longer exists
Complete and abort both remove .uploads/<id>. abortMultipartUpload answered
204 whenever that record was missing, so aborting an upload that had already
completed reported success while the object stayed in place, and a client
could read it as the object being discarded. AWS returns 404 NoSuchUpload
("The specified multipart upload does not exist") for that case, and
ListParts here already does.
Answer ErrNoSuchUpload when the record is not found. checkUploadStillOpen
reuses abort for cleanup after the upload is gone; NoSuchUpload is not a
failed cleanup there.
The leftover-directory path from #11385 is unchanged: a .uploads/<id> that
outlived its completed object is still removed metadata-only and answered.
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* volume server: add last-entry cursor fields to VolumeIncrementalCopy
* backup: resume incremental copies by index position instead of appendAtNs
appendAtNs is stamped by whichever replica received the write, so the
same needle carries a different timestamp on every replica. A backup
that resumes from a different replica than last time can append the
last needle again or skip needles written inside the clock gap.
Send the local copy's index entry count and last entry identity (key,
size, cookie) with VolumeIncrementalCopy. The replica verifies the same
record sits at that position and streams from the next index entry, so
the resume no longer depends on any replica's clock. When the boundary
record does not match, or the local copy is ahead, the replica returns
FailedPrecondition and the backup rebuilds the local copy from scratch.
Old servers ignore the new fields and keep the timestamp behavior.
* rust volume server: mirror positional resume in VolumeIncrementalCopy
* volume server: add entries_hash prefix checksum to VolumeIncrementalCopy
* backup: verify the whole index prefix, not just the boundary entry
* rust volume server: verify the index prefix hash in resume_by_last_entry
* backup: keep the source's compaction revision on rebuilt copies
* backup: diverge on a zero-offset boundary entry
A remote-tier tombstone lives in the .idx with no .dat record, so
readNeedleAt(0) parses the superblock and its error bypassed the
ErrIncrementalCopyDiverged mapping; the backup got a generic failure
instead of FailedPrecondition and never rebuilt.
* rust volume server: mirror zero-offset boundary divergence
* rust volume server: gate the empty-dat early return on entry_count
With a positional resume the Go handler still reaches ResumeByLastEntry,
so an empty replica reports divergence; the Rust early return answered a
clean empty stream and weed backup kept a stale copy.
* backup: hash the index entry's size field, not trailing offset bytes
In a 5BytesOffset build the offset occupies bytes 8-12 and the size
starts at byte 13, so the hash folded in one byte of the offset.
Matching records at different offsets then hashed differently and a
valid backup was reported diverged and rebuilt.
* backup: read the index in whole-entry chunks for the prefix hash
When entries are 17 bytes a 1 MiB read buffer does not end on an entry
boundary: the partial tail entry was skipped by the loop but still
counted as consumed, so every later chunk was hashed at a misaligned
position.
* rust volume server: hash the index entry's size field after the offset
Mirrors the Go hashIndexEntries change: with the 5bytes feature the
offset occupies bytes 8-12, so starting the size slice at byte 12
folded one offset byte into the hash and matching records at different
offsets compared diverged.
* rust volume server: read the index in whole-entry chunks for the prefix hash
Mirrors the Go change. With 17-byte entries a 1 MiB buffer does not
end on an entry boundary, so slicing the partial tail entry panicked
once a backup held 61681 entries.
* rust volume server: default the new incremental copy request fields in tests
* test: align vector bucket policy coverage with vector actions
* s3vectors: validate ARN service and resource shape before resolving indexes
* test: exercise signed vector bucket policies and catalog lifecycle
* ci: require the expanded vector integration matrix and all gateway tests
* test: reject index ARNs in bucket-level API requests
* s3vectors: scope ARN resolution to the gateway namespace
* s3vectors: reject name fields that conflict with resolved ARNs
bucketARNs and resolveIndexARNs already reject an indexArn that names a
different bucket than vectorBucketArn, but a vectorBucketName or
indexName disagreeing with the ARNs was silently dropped. Validate the
plain names against the resolved ARN values the same way.
* test: cover conflicting vector names and ARNs in live requests
---------
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* s3vectors: reject empty boolean-operator lists and sub-objects in filters
An empty $and/$or list or a nested empty object rendered '()'' — invalid
SQL surfaced as an engine error instead of a clean ValidationException.
* s3vectors: e2e conformance for vector bucket policy evaluation
The handler-level test covers statements; this runs the whole stack a
client sees — HTTP dispatch, authentication, policy evaluation, catalog,
engine — and walks the full lifecycle: a non-owner is denied without a
grant, an Allow on the bucket ARN grants every index under it, a Deny
vetoes an Allow on the same action, and deleting the policy restores
owner-only access.
* s3vectors: prove deny precedence and grant removal in policy e2e
* s3vectors: reject field operator objects with no operators
* test: use an s3vectors action in the vector policy round-trip
* s3vectors: reject empty boolean-operator lists and sub-objects in filters
An empty $and/$or list or a nested empty object rendered '()'' — invalid
SQL surfaced as an engine error instead of a clean ValidationException.
* s3vectors: reject field operator objects with no operators
* s3api: label bucket-level requests correctly in the access log
getResourceType treated a request as bucket-level only when the object key
was "/". NormalizeObjectKey strips the leading slash, so bucket routes arrive
with an empty key and every bucket operation was logged as an object
operation: ListObjectsV2 as REST.GET.OBJECT, DeleteObjects as
REST.POST.OBJECT, and the lifecycle, ACL and policy calls likewise.
Treat an empty object key as bucket-level too. The new test covers five
bucket-level and two object-level request shapes; it fails without the fix.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
* s3api: label only POST ?delete as a batch delete in the access log
With bucket-level requests now reaching the bucket branch, any method on
/bucket?delete was labelled BATCH.DELETE.OBJECT. Only the POST route is
DeleteObjects; label other methods as the plain bucket operation.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
* mount: refuse to attach opens to a draining file handle
With the writeback cache, releasing the last open of a file leaves the
shared FileHandle in the map at counter zero while an async flush
drains it; the worker removes the map entries once the flush commits.
AcquireHandle waits for a pending flush and then attaches, but the two
steps are not atomic: a release can slip in between, drop the counter
to zero, and queue the drain while the opener goes on to mint a new
file-handle id on the draining handle. When the drain completes,
RemoveFileHandle deletes every fh id for the inode -- including the one
just minted. The kernel still holds that id, so its next FUSE_READ
gets ENOENT from GetHandle; the failed read leaves the folio not
uptodate and the kernel surfaces EIO to userspace. This was the flaky
TestWritebackCacheConcurrentMixedOps failure.
AcquireFileHandle now refuses to attach when the shared handle has
reached a zero counter and reports it as draining. AcquireHandle and
the Create path retry the whole wait-resolve-attach sequence; the next
pass blocks on the pending flush (flag registration and the counter
drop share pendingAsyncFlushMu) and builds a fresh handle from the
post-flush entry. The DLM-failure release in AcquireHandle also goes
through ReleaseHandle so it cannot strand a counter-zero handle
outside the drain bookkeeping.
Removes the refreshEntryForRead retry from #12206, which worked around
the symptom by refetching the entry after the fh id was already gone.
Test: TestAcquireFileHandleRefusesDrainingHandle covers the refusal
and fresh-handle path; TestWritebackCacheConcurrentMixedOps and an
amplified concurrent-overwrite repro run clean.
* mount: keep refresh-on-error retry for stale-chunk reads
* s3vectors: bound ListVectorBuckets and ListIndexes maxResults
Both pass maxResults to the catalog unbounded; AWS caps these list pages
at 100, the same way ListVectors is bounded and topK is bounded.
* s3vectors: raise ListVectorBuckets/ListIndexes maxResults cap to 500
AWS documents a valid range of 1-500 for both list APIs; the shared
limit of 100 rejected legitimate page sizes.
* s3vectors: reject explicit maxResults of zero on list calls
AWS requires maxResults >= 1, but a plain int field cannot tell an
omitted field from an explicit zero. Making MaxResults a pointer keeps
the catalog default for omitted fields while a supplied zero fails
validation.
* s3vectors: test max ListIndexes page against an existing bucket
* s3tables: reject s3vectors resources in bucket policies
An arn:aws:s3vectors: Resource is as dead as an s3vectors action: the
evaluator compares against the generated s3tables ARN, so the rule the
writer believes enforced never matches.
* s3tables: reject s3vectors resources in every ARN partition
* s3tables: test wildcard-partition and mixed-list s3vectors resources
* filer.sync: skip superseded replays before fetching any chunks
On a directory under heavy overwrite, most entries the sync replays are
already superseded on the source. The sink only learned that after chunk
reads failed — or worse, after fetching and uploading bytes a later event
replaces — so the backlog drained slowly and the log filled with 404s.
CreateEntry and UpdateEntry now check sourceMovedOn before touching a
chunk: one entry lookup skips a stale replay entirely. A mid-replication
supersession is still caught by the existing retry gates.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* replication: bound the source liveness probes with a timeout
The supersession checks added a source entry lookup to the replication
path, and every lookup ran on context.Background() with no deadline. A
source that accepts the call but whose metadata store stalls would hold
the sync forever — before a single chunk fetch, even when every chunk is
still readable. Bound the probe at 30s; an unanswered lookup proves
nothing, so replication proceeds and the existing retry gates decide.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* replication: sync the live version a superseded replay points at
Skipping a stale replay assumed a later event always re-carries the
current content. It does not when the superseding write was itself
replicated in from another cluster: those events are dropped before they
reach a filer sink, so the destination could be left missing the file.
When the source reports a live entry newer than the replay, commit that
version through the normal path instead — equivalent to applying the
later event, with no dependence on its delivery. A gone source entry
still skips, since the destination already matches the source.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* replication: reconcile moved entry content after a native rename
AtomicRenameEntry relocates whatever version the destination held. If the
preceding update replay was skipped because the source had already moved
to the new name, the moved entry keeps the old bytes and — with no
further writes — stays stale forever. MoveEntry now runs the same
reconciliation as an in-place update against the rename's snapshot, so
the moved entry picks up pending content changes.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* replication: repair from the live snapshot a superseded replay resolves to
An incremental diff against the replayed oldEntry can resolve manifests the
overwrite already vacuumed, which fails every retry of the event.
---------
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* s3vectors: translate vector bucket policies to the catalog form
The document is stored verbatim on the shared table bucket and evaluated
against s3tables actions and ARNs, so a policy written for the vector API
was a dead rule. Actions now map to the catalog operation the request
authorizes against, and an arn:aws:s3vectors: resource becomes the
bucket/.../table/vector/... ARN the index's catalog entry answers to.
* s3tables: expose PrincipalFor and PolicyContext for façade evaluation
A façade authorizing against its own action namespace needs the same
identity resolution and evaluation context the catalog uses internally.
* s3vectors: keep bucket policies in vector form and evaluate them in the gateway
The earlier approach translated the document to catalog actions and ARNs,
which lost the caller's original policy for GetVectorBucketPolicy and could
not cover data-plane calls that never touch the catalog. The document now
stays in s3vectors form under a dedicated bucket attribute and every
handler evaluates it through authorizeVector against the action and ARN
the request names; a dedicated key also keeps a vector grant from ever
authorizing a catalog call.
* s3vectors: close policy-mutation races and keep vector grants visible
- Authorize Put/DeleteVectorBucketPolicy inside the policy slot's
compare-and-swap, so a grant revoked between reads cannot authorize
the pending write.
- Assert the policy attributes stay absent in the update precondition,
so a concurrent policy or tag write trips the retry instead of being
silently reverted.
- Migrate a raw vector document out of the shared policy slot on
replace or delete, keeping its catalog rules.
- Pass the unresolved identity to PolicyContext so open-by-default
deployments still work.
- Filter ListVectorBuckets on the vector policy itself, so buckets a
caller was granted under s3vectors actions stay listed.
* s3vectors: reuse SnapshotExtended for the vector-policy CAS precondition
* s3vectors: skip buckets that vanish mid-list instead of failing the page
* s3vectors: reject non-vector bucket formats inside authorizeVector
* s3vectors: check bucket format after authorization in authorizeVector
An unauthorized caller probing an ICEBERG bucket got NoSuchBucket,\nleaking that the bucket exists. The format check now runs after the\npermission check so denied callers always see AccessDenied.
* s3vectors: cover legacy shared-slot policy removal on delete
* s3tables: resolve the forwarded identity name like getAccountID in PrincipalFor
A synthesized manager request resolves its caller through the identity-
name branch of getAccountID, which normalizes the name (ARNs collapse to
their trailing segment, whitespace trims) before falling back to the
handler's account ID. PrincipalFor returned the raw name, so ARN-shaped
identities never matched their own grants.
* s3tables: assert the vector policy key in update preconditions
s3vectors-policy is stored on table bucket entries but was missing from
s3tablesExtendedKeys, so a catalog update that read the entry before the
first vector policy was saved could overwrite it without tripping the
compare-and-swap. Asserting the key makes such a race fail instead of
silently erasing the policy.
* s3vectors: reject rules the evaluator cannot honor in vector bucket policies
The shared Statement shape has no negated fields, the vector evaluation
supplies no condition keys, and a document that fails the typed decode
is discarded wholesale at evaluation — including denies the writer
believed active. Reject NotPrincipal, Condition, foreign-service actions
and resource ARNs, and require the typed PolicyDocument decode. Also
cover that an admin-named identity still reaches its own buckets.
* s3vectors: read vector policy fields the way the evaluator decodes them
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* s3tables: match ARN policy principals through caller normalization
Caller principals are normalized ids, so a statement naming a full IAM
ARN could never match: an ARN-scoped deny was silently dead and an
ARN-scoped allow unreachable. Compare arn: specs against the caller
under the same normalizePrincipalID the ownership path uses, while an
ARN caller keeps verbatim wildcard matching.
* s3tables: grant the admin bypass on the caller's actions, not the name
A principal that only normalizes to the admin id skipped every policy
deny. When the request carries an identity, the bypass now requires the
identity's own admin action — the same gate getAccountID applies — while
name-only transports with no action list keep it. PolicyContext takes
the request so it can carry IdentityActions, and the vector context
helpers route through the shared identity keys so the resolved name and
actions agree.
* s3tables: match ARN policy principals only for deny statements
Normalization drops the ARN's account and path, so an allow matched on the
trailing name could grant a same-named caller the statement never verified.
Denies keep the normalized match so the rule cannot be ducked through the
same collapse.
* s3tables: scope service wildcards to their own service
s3tables:* counted as blanket admin inside the shared permission check, so
an identity holding only catalog grants administered s3vectors operations
too. hasAdminActionFor now requires the wildcard to name the operation's
service; the global admin markers and s3:* stay service-agnostic.
---------
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* s3vectors: reuse shared authenticated identity context
* s3vectors: encode empty results as arrays
* test: cover S3 Vectors with Go and Rust volume servers
* ci: require both volume implementations in vector integration tests
* s3vectors: keep identityOf as a shim over the shared identity context
Concurrent branches add identityOf call sites; deleting the helper would
break the build when they merge in either order. Route it through
GetIdentityNameFromContext so every caller resolves the same identity.
* test: wait for the vector engine to register before running
The gateway resolves weed-vector lazily through the master cluster list,
so a listening port does not mean the engine is reachable yet; the first
vector write or query could race KeepConnected and fail. Restore the
cluster.ps registration wait the previous harness had, and ignore the
pytest cache directory the suite creates.
* test: wait for master registration across OSS and enterprise
Use the exact engine registration event after AddClusterNode completes. The current enterprise cluster.ps does not list vector nodes, so waiting on its output prevents the shared fixture from starting there.
* s3vectors: keep contextWithIdentity as a shim alongside identityOf
In-flight callers still write the caller name through it; route the
write through SetIdentityNameInContext so both helpers agree on the
shared key whichever lands first.
An arn:aws:s3vectors: Resource is as dead as an s3vectors action: the
evaluator compares against the generated s3tables ARN, so the rule the
writer believes enforced never matches.
* vector: implement drop_index in the engine
The RPC was a stub; it now resolves the table's vended location and
removes every object under it through the same credential path open uses.
A table the catalog can no longer describe is already dropped, so the call
stays idempotent.
* s3vectors: drop index data after the catalog delete
DeleteTable carries the delete authorization, so it runs first; the engine
DropIndex then removes the data under the index's location. NotFound and
Unimplemented are tolerated so an index whose data went away with the
catalog entry, or an older engine, still answers 200.
* s3vectors: let the catalog delete carry the index data removal
A vector index is colocated: the catalog entry sits at the dataset root,
so DeleteTable's recursive delete already removes entry and data. The
engine DropIndex call afterwards could only ever describe a missing or
recreated record — a no-op in the common case, and a delete of a
replacement index's files when the name was reused in between.
* filer_pb,filer: per-mutation preconditions on ObjectTransaction
A transaction's request-level condition evaluates one entry before any
mutation runs, so a commit guarding two paths — the object it publishes
and the upload directory it assembles — could not be expressed. Give each
ObjectMutation an optional condition (evaluated against condition_key,
else the mutation's target, under the same lock just before the mutation
applies) and report which mutation failed through failed_mutation on the
response, so callers can tell a mutation guard's rejection apart from the
request-level precondition's.
* s3: evaluate multipart completion conditions under the object lock
CompleteMultipartUpload checked conditional headers gateway-side before
committing, and the routed commit transaction guarded only the upload
directory — nothing made If-None-Match: * atomic with the publish, so
concurrent conditional completions of the same absent key could all
report success.
The request's reducible write condition (If-None-Match/If-Match, star or
single strong ETag) now rides the commit as the transaction's request
condition on the owner filer: on the object path for unversioned buckets,
falling back to the .versions directory for suspended and enabled
versioning. The upload-exists guard moves onto the PUT mutation, keeping
its check-before-publish ordering. Conditions the reducer cannot express
stay on the distributed write lock, where check and commit already
serialize.
* filer,s3: resolve the latest version first for versioned conditional writes
* filer: keep directory entries holding content out of the absent-slot rule
* s3: keep a gateway-side upload-exists check for filers without mutation conditions
* filer: honor null-version-is-latest resolving prefer-alt conditions
* drop scratch multipart-race repro programs
* s3: note the write lock does not cover routed writes on non-reducible conditions
* filer: mirror the gateway's remote-etag precedence in condition evaluation
storedEntryETag only knew the extended attribute and the chunk fallback,
so a synced remote entry compared chunk-derived ETags while the gateway
compares the remote ETag. Follow the same precedence: extended attribute,
remote ETag while the entry still holds the remote bytes, Md5, then the
chunk hash — and no ETag for an entry holding none.
* s3: reduce all conditional write headers into owner-lock clauses
Time-based headers, combined conditions, ETag lists, and weak ETags stayed
on the distributed write lock because buildWriteCondition could not express
them, leaving their gateway check racing owner-routed writes. Reduce every
parseable header form to WriteCondition clauses — multiple clauses compose
AND, ETag candidates carry the raw list for the filer's canonicalization,
and RFC 7232 precedence keeps If-Unmodified-Since hidden by If-Match and
If-Modified-Since hidden by If-None-Match — so the owner filer evaluates
them under the same object-path lock as the commit. Only unparseable
headers still fall back, where the lock path reports the same parse error.
* filer_pb,filer: per-mutation preconditions on ObjectTransaction
A transaction's request-level condition evaluates one entry before any
mutation runs, so a commit guarding two paths — the object it publishes
and the upload directory it assembles — could not be expressed. Give each
ObjectMutation an optional condition (evaluated against condition_key,
else the mutation's target, under the same lock just before the mutation
applies) and report which mutation failed through failed_mutation on the
response, so callers can tell a mutation guard's rejection apart from the
request-level precondition's.
* s3: evaluate multipart completion conditions under the object lock
CompleteMultipartUpload checked conditional headers gateway-side before
committing, and the routed commit transaction guarded only the upload
directory — nothing made If-None-Match: * atomic with the publish, so
concurrent conditional completions of the same absent key could all
report success.
The request's reducible write condition (If-None-Match/If-Match, star or
single strong ETag) now rides the commit as the transaction's request
condition on the owner filer: on the object path for unversioned buckets,
falling back to the .versions directory for suspended and enabled
versioning. The upload-exists guard moves onto the PUT mutation, keeping
its check-before-publish ordering. Conditions the reducer cannot express
stay on the distributed write lock, where check and commit already
serialize.
* filer,s3: resolve the latest version first for versioned conditional writes
* filer: keep directory entries holding content out of the absent-slot rule
* s3: keep a gateway-side upload-exists check for filers without mutation conditions
* filer: honor null-version-is-latest resolving prefer-alt conditions
* drop scratch multipart-race repro programs
* s3: note the write lock does not cover routed writes on non-reducible conditions
* mount: keep the shared directory build out of the requester's context
The shared directory build behind EnsureVisited is a singleflight call: the
first requester for a path owns the build, and every later requester for the
same path waits on that same build. The build ran on the owner's request
context, so a FUSE request that returned -- or that the kernel interrupted --
canceled the build for everybody waiting on it.
A waiter whose own context was still alive then received somebody else's
cancellation. weed/mount/weedfs_dir_read.go classified a canceled read as
EINTR only when the local context was done, so such a waiter fell through to
fuse.EIO: the kernel handed EIO to readdirent, kubelet's applyFSGroup failed,
and the volume never mounted.
Build the shared directory on a context that keeps the request's values but
not its cancellation, and report a canceled read as EINTR even when the local
context is still alive. Waiters still abandon on their own context, so a
canceled request stops waiting immediately.
This follows up #11717, which made the wait abandonable and mapped the local
cancellation to EINTR, but left the shared build itself on the owner's
context. The surrounding build lifecycle calls already use
context.Background() for the same reason.
Test: TestEnsureVisitedSharedBuildSurvivesOwnerRequestCancel reproduces the
cancellation with a fake filer and fails without this change.
* mount: classify gRPC-canceled directory listings as EINTR
* mount: log only real directory read failures, not interruptions
* mount: keep shared section refresh out of the requester's context
* mount: let section-refresh waiters leave on their own context
The shared refresh flight now outlives the requester that started it,
but callers still blocked in singleflight.Do until it finished — an
interrupted directory read could not stop waiting, and the caller's
own deadline never applied to the wait. DoChan + select on the
caller's context restores early abandon while the detached refresh
still completes for the remaining waiters.
* ci: re-run FUSE integration after a flaky writeback-cache failure
* mount: refresh a stale entry once before reporting a read error
A read resolves chunk views from the handle's entry. A concurrent
overwrite commits a new entry whose replaced file ids the filer queues
for deletion; the open handle only learns the new chunk list when the
metadata invalidation lands. A fetch that lands between the volume
delete and the invalidation names data that is already gone and the
read reports EIO. On a chunk read error, wait out any in-flight async
flush, re-read the entry from the filer, install it when it outranks
the handle, and retry once. An unchanged version means the failure is
genuine.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* mount: fence the read-error entry refresh by file identity
Two guards on refreshEntryForRead's install path: the unlocked
dirtyMetadata read raced with Write under the handle lock — drop it,
installAckedEntry already checks under that lock, and it now fences
isDeleted handles the same way. And a saved path can name a different
file after unlink/reuse, so refuse the refresh when the fetched inode
disagrees with the one the handle holds.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: user.email <chrislusf@users.noreply.github.com>
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* s3tables: carry encryptionConfiguration in table bucket metadata
* s3vectors: accept and return encryptionConfiguration on vector buckets
The configuration is validated (sseType AES256 or aws:kms) and stored on
the bucket record, so the API round-trips what the caller declares.
* s3vectors: reject caller-chosen encryption keys on vector buckets
* s3tables: reject caller-chosen encryption keys on table buckets
* s3tables: reject unknown sseType in bucket encryptionConfiguration
The bucket check rejected aws:kms and kmsKeyArn but stored any other
sseType — a made-up or mistyped value would round-trip through
GetTableBucket as an encryption claim the storage does not keep. The
validator now admits only the AES256 declaration, shared so the vector
layer can drop its copy.
* s3vectors: delegate encryption validation to the shared catalog check
* s3tables: accept the sseAlgorithm name in bucket encryptionConfiguration
S3 Tables declares the algorithm as sseAlgorithm while S3 Vectors uses
sseType; the shared check now accepts either spelling, still admits only
AES256, and still refuses any caller-chosen key.
* s3vectors: call the shared encryption validator directly
* s3vectors: require each API's own encryption field name
The shared validator accepted sseAlgorithm on the vector path, so a
CreateVectorBucket carrying the s3tables field stored a configuration
GetVectorBucket could never echo back in the vectors shape (and vice
versa). ValidateEncryptionConfiguration now takes the wire name its
caller expects; the vectors create path normalizes to sseAlgorithm for
the catalog and GetVectorBucket renames it back to sseType.
* s3vectors: bound QueryVectors topK at the AWS limit of 30
topK passed through unbounded; AWS caps a query at 30 results, so a
larger request is a ValidationException before it reaches the engine.
* s3vectors: raise QueryVectors topK bound to the AWS limit of 10,000
* s3vectors: validate QueryVectors topK before the catalog lookup
* s3vectors: cover the QueryVectors topK bounds
* util: do not bind outbound dials to a loopback -ip.bind
`weed server -ip.bind=127.0.0.1` keeps every listener local, but the
same address became the source of outbound connections, so the filer
dialled its remote mount from 127.0.0.1 and failed with "connect:
invalid argument". A loopback source cannot reach off the host; leave
the source to the OS, as for a wildcard bind.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3vectors: put, get and delete vector bucket policies
The AWS S3 Vectors policy operations had no handlers; the policy document
is stored on the shared table bucket, so the handlers delegate to the
catalog's PutTableBucketPolicy, GetTableBucketPolicy and
DeleteTableBucketPolicy and keep its guard.
* s3vectors: treat ICEBERG buckets as absent in policy handlers
GetVectorBucketPolicy/PutVectorBucketPolicy/DeleteVectorBucketPolicy
delegated straight to the table bucket, so a bucket created for ICEBERG
tables — invisible to every other vectors operation — could still have
its policy read and written through the vectors API. Gate all three on
lanceBucket, the same format check DeleteVectorBucket and CreateIndex
already apply.
* test/vectors: cover vector bucket policy put/get/delete
Round-trips a policy document through the new ops and asserts a get on
the deleted policy surfaces NotFoundException, so the boto3 conformance
run exercises the added handlers end to end.
* s3vectors: validate the bucket ARN and policy document
Bucket-level operations now reject an index ARN and an ARN minted for
another service, and PutVectorBucketPolicy refuses a document carrying
s3vectors actions — the evaluator only honors s3tables actions, so such a
rule would be a dead grant.
* s3vectors: validate the bucket ARN on the remaining bucket-level operations
* s3vectors: cover bucket ARN and policy document validation
TestBucketARNs pins the accepted forms and the index/foreign-service
rejections; TestValidatePolicyDocument covers a valid document, a
malformed one, and the dead s3vectors action namespace.
* s3vectors test: use an enforceable action in the bucket policy test
The catalog refuses s3vectors actions in a stored policy, so the put call
rejected the document before the get and delete assertions ran.
---------
Co-authored-by: Ólafur Bjarki Bogason <olafur@genkiinstruments.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* s3vectors: cap PutVectors at 500 items and check data against the index dimension
A batch could carry any number of vectors and a vector of the wrong
dimensionality reached the engine: AWS caps a PutVectors call at 500 items
and requires each vector to match the index's declared dimension.
* s3vectors: cover the PutVectors batch and dimension checks
* s3vectors: reject query vectors that do not match the index dimension
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* s3vectors: cover the PutVectors batch size bounds
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* s3vectors: cover the catalog-backed PutVectors handler path
Run PutVectors through the real handlers over the in-memory catalog: a
matching vector reaches the engine, a mismatching one is rejected with
ValidationException before PutVectors is called, and a full
vectorsMaxPutBatch batch is accepted.
---------
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* iceberg maintenance: an unreadable parquet file fails the job, not the process
parquet.NewReader panics on any file it cannot open (wrong magic, a
truncated footer, zero bytes), and parquet-go's row decoder panics on a
corrupt page under a valid footer. Maintenance runs in the worker's
goroutines with no recover, so one bad delete or data file in any table
ended the whole process.
- Every parquet open in compaction goes through newParquetReader, which
opens with parquet.OpenFile and returns the error.
- A panic while decoding a delete file or a data file's rows is recovered
into an error naming the file (recoverRead/guardRead).
- As a backstop, detection recovers per table and Execute per operation:
the table is skipped or the operation fails, and the worker keeps
running.
A failed delete file still fails the table's compaction (no commit), as a
read error did before; a failed data file still skips only its bin.
* iceberg maintenance: cache a remote-only file before reading it
On a bucket mounted with remote.mount, a file that is not cached locally
has no chunks and no inline content, only its RemoteEntry. Maintenance
read it as an empty file: a parquet reader failed on zero bytes (before
the previous change, it panicked and ended the process) and a manifest
failed to parse, so a table whose files had been uncached could not be
maintained.
lookupFileByIcebergPath, the lookup every read goes through, now calls
CacheRemoteObjectToLocalCluster for a remote-only entry, as the S3
gateway does on a read, and reads the entry it returns; the call is tried
twice, and an entry still remote-only after caching is an error. The
lookups that only resolve a path (orphan removal) do not cache.
readEntryContent reports a file with no content and no chunks as empty
instead of returning zero bytes.
The fake volume server in the tests now serves chunk reads, so the
remote-only tests read the cached file through the chunk reader as a
real one is read.
* iceberg maintenance: bound the metadata location read before caching a remote file
* iceberg maintenance: bound the metadata read on the entry the cache returns
* iceberg maintenance: enforce the metadata size bound inside the remote cache call
The worker checks the resolved entry before asking the filer to cache a
remote-only object, but the filer resolves the entry again at cache time
and downloads whatever its remote size then says. A replacement landing in
between could push an oversized object through the download even though
the post-cache check would refuse to read it.
CacheRemoteObjectToLocalClusterRequest gains an optional max_size; the
filer rejects an over-limit remote object before the download starts.
loadMetadata passes metadataFileMaxBytes; data-file reads leave it unset.
* iceberg maintenance: key the remote cache singleflight by the size limit
Callers sharing one flight share its outcome, including a bounded
caller's over-limit rejection: an unrestricted reader joining the same
path would fail even though its own bound allows the download. The
limit joins the key so only callers with the same bound share a flight.
* iceberg: gofmt compact_order_test.go
* filer: bound remote cache callers before the shared download
Each caller's max_size is now enforced on the entry the filer resolves
before that caller joins the singleflight, which is keyed by path again.
A bounded caller's rejection is never shared with other callers, and
callers with different bounds share one download instead of pulling the
same remote object twice.
* filer: bind the remote cache size bound to the downloaded entry
The pre-flight check and the shared download resolve the entry
separately, so a remote metadata refresh landing in between lets a
larger object through: the download ran to the refreshed size and only
the reader's post-check refused it. The bound is now re-checked inside
the flight on the entry the download streams.
The rejection is typed so a co-tenant sharing the flight does not
inherit it: callers whose own bound permits the reported size retry on
a fresh flight instead of failing for another caller's limit.
---------
Co-authored-by: user.email <chrislusf@users.noreply.github.com>
* mount, s3: clarify allowOthers requirement and bucket name error
`weed mount -allowOthers` defaults to true. On Linux, a non-root
user's mount then needs `user_allow_other` in /etc/fuse.conf, but the
help text only said "allows other users to access the file system".
Say so in the help, and point at -allowOthers=false.
Also fix the grammar of the bucket name length error ("must between
[3, 63]" -> "must be between 3 and 63").
Help and error text only; no behaviour change.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3: normalize the remaining VerifyS3BucketName error messages
The length message was clarified; the rest of the function still had
inconsistent phrasing ("name must start", "cannot be ip addresses",
"prefix xn-- ... in bucket prefix"). Align them on "bucket name ..."
wording matching the AWS bucket naming rules.
---------
Co-authored-by: pingqiu <pingqiu999@gmail.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: user.email <chrislusf@users.noreply.github.com>
`weed server -ip.bind=127.0.0.1` keeps every listener local, but the
same address became the source of outbound connections, so the filer
dialled its remote mount from 127.0.0.1 and failed with "connect:
invalid argument". A loopback source cannot reach off the host; leave
the source to the OS, as for a wildcard bind.
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
The vectors gateway backs off discovery for 5s between resolutions, but
the backoff also suppressed rediscovery after an empty lookup. A worker
that landed its KeepConnected registration a moment after the first
query stayed invisible for the whole window — longer than a client's
retry budget — so the S3 Vectors e2e flaked: CreateIndex raced the
registration, got InternalFailure, and every boto retry fell inside the
same 5s fast-fail window.
With no cached backend, recheck every 500ms instead; the 5s interval now
only bounds membership refreshes against a live client, and an absent
worker still cannot flood the master with lookups. The empty-membership
error also becomes codes.Unavailable so the API answers 503 rather than
InternalFailure.
cluster.ps lists registered vector nodes like the other node types, and
the e2e harness polls it for the engine's registration instead of only
checking the gRPC port — a listening engine is not discoverable until
the registration lands.
* volume: remount ec shards the rebuild replaced under a mounted handle
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* ec.rebuild: keep copied shards the rebuild regenerated in place
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* volume: reopen regenerated ec shards in place instead of remounting
UnmountEcShards and MountEcShards send deltas through separate buffered
channels, so a remount can reach the master as add-then-delete and drop
the repaired shard from lookups until the next full EC heartbeat.
ReopenEcShard swaps the mounted shard's descriptor under the volume lock
instead: registration, disk type, and heartbeat state never flip, and
readers holding the old shard degrade to a clean error the parity path
already recovers from.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* rust volume: reopen rebuilt ec shards in place, mirroring Go
volume_ec_shards_rebuild recreates a shard file under a mounted handle
whose descriptor still serves the detached inode (a mounted shard whose
file was deleted under it). DiskLocation::reopen_ec_shard swaps the
mounted descriptor for a fresh open, matching Go's ReopenEcShard — the
registration and heartbeat view never flip.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* volume: fail ec rebuild when a regenerated shard cannot be reopened
* rust volume: fail ec shards rebuild when a rebuilt shard cannot be reopened
* volume: repoint mounted ec shards after rebuild across disks
A mounted shard whose file was deleted under the mount is "missing" to
the rebuild: the regenerated file lands on the rebuild location while
the handle keeps serving the detached inode on another disk. Reopening
the mounted disk then failed the whole RPC, so the caller never saw the
rebuilt ids and its deferred cleanup deleted the regenerated copies.
Repoint every mounted handle of the rebuilt volume instead of only this
run's outputs: reopen in place when the rebuilt file is on the mounted
disk, otherwise drop the stale handle and remount the shard from
whichever local disk holds the file via a delta-free load split out of
MountEcShards — a same-server move, so the master's registration never
flips. A shard that can be neither reopened nor remounted is left
unmounted rather than serving replaced bytes, and the repoint step no
longer fails the RPC, so RebuiltShardIds always reaches the caller.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* rust volume: repoint mounted ec shards after rebuild across disks
Mirror the Go repoint loop: iterate every mounted shard of the rebuilt
volume, reopen in place when the rebuilt file is on the mounted disk,
and otherwise drop the stale handle and remount the shard from
whichever local disk holds it. The repoint no longer fails the RPC, so
rebuilt_shard_ids still reaches the caller and its copy cleanup keeps
the regenerated shards.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* rust volume: still rebuild ecx and repoint mounts when no shard is missing
* master: keep a node in an ec shard's lookup while a disk still holds it
A rebuild that relocates a mounted shard between a node's disks reports
the move as a delete on the source disk plus an add on the destination.
Registering first is a no-op because shard lookups key on the node, then
the unregister dropped the node outright — and steady-state heartbeats
never restore it, so readers and later rebuilds lose the repaired copy.
Skip the delete while another disk on the node still holds the shard.
UnregisterDataNode now clears the node's own ec records before dropping
its registrations, so the retained-shard check cannot skip the teardown,
and each unregister gets a ShardsInfo copy since DeleteEcShard drains
the shared one.
* rust volume: fold the journal on a mounted index instead of rewriting it
The rebuild handler regenerated .ecx through File::create even while the
volume was mounted, truncating the index under readers that search the
open descriptor with its old size — a live lookup could hit EOF mid-scan.
Go folds the deletion journal in place on a mounted volume; do the same
through rebuild_ecx_from_journal and reserve the full regen for the
unmounted path, where nothing holds the file open to interrupt.
* volume, rust volume: report rebuilt shards on late failures, fix rust corrupt discovery
Two review findings the repoint rework left open:
- A RebuildEcxFile failure still returns a plain error after shards were
regenerated, so the caller discards the response holding the rebuilt
ids and the deferred cleanup deletes copies the rebuild regenerated
in place. The Go handler now packs the ids into the gRPC status
details and the shell pulls them back out on error; the Rust handler
carries them in grpc-status-details-bin the same way.
- The Rust rebuild still counted a sidecar-corrupt shard as present, so
it fed bad bytes into reconstruction and never regenerated the shard.
Discovery now loads the generation-0 .ecsum the way Go does, verifies
every present shard, and reclassifies mismatches (and zero-size
residue) as rebuild targets written to a .rebuilding temp file,
verified, and atomically renamed over the original only after every
output passes the sidecar check. Honors unsafe_ignore_sidecar, added
to VolumeEcShardsRebuildRequest to match the Go proto.
* volume: repoint sibling journals sharing a folded .ecj after rebuild
RebuildEcxFile unlinks the deletion journal and the mounted path only
reopens its own handle — a sibling mount journaling into the same file
keeps appending to the detached inode, and those deletes never reach a
path-based reader again. Reopen every mounted volume sharing the index
base before leaving the rebuild.
* rust volume: repoint sibling journals sharing a folded .ecj after rebuild
rebuild_ecx_from_journal replaces the journal and reopens only its own
handle; mounts on sibling disks sharing that .ecj keep the detached
handle, so their next appends are lost at restart. Adopt the folded
journal on every mounted runtime sharing the path before releasing the
store lock, clearing their in-memory set — every journaled id is now an
.ecx tombstone.
* ec: swap the deletion journal handle only after the new file opens
A failed reopen left ecjFile nil, so every later delete returned
"ec volume closed". Open first, then close and swap — mirroring the
Rust adopt_folded_journal ordering.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* ec.rebuild: fail the rebuild when a sibling journal cannot be repointed
A warning left the sibling mount journaling to a detached inode while
the RPC reported success. Propagate the error like the Rust handler's
adopt_folded_journal map_err.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* ec: keep deletes loud when the journal reopen cannot open the replacement
Reverts the swap-on-success ordering: a detached handle kept on failure
lets deletes append and sync to the removed inode, then vanish at
restart. Closing first also frees the descriptor before the open.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* rust volume: drop the folded journal handle before reopening it
adopt_folded_journal and the post-fold recreate now clear ecj_file
before opening the replacement, so a failed reopen leaves deletes
erroring instead of appending to the detached inode. Mirrors Go's
ReopenDeletionJournal.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* ec: drop the cached journal size when its reopen fails
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* remote: reuse one guarded remote storage client per configuration
BuildGuardedRemoteStorageClient (the filer's lazy remote stat and list,
remote deletes and renames, the volume server's cold reads, the S3
gateway's remote reads) built a new client, with its own http.Transport,
on every call. Each transport kept its connection idle for
IdleConnTimeout (60s) and was never reused or closed, so a steady stream
of remote lookups held one open connection per request. An endpoint
that caps connections per source then refuses every new connection.
- remote_storage: GetRemoteStorage's per-name cache becomes a bounded
LRU (WEED_REMOTE_CLIENT_CACHE_MAX, default 64) keyed by kind, name, a
hash of the whole configuration and a credentials version, built
outside the lock once per key (singleflight) from a copy of the
configuration. A client leaving the cache has its idle connections
closed. GetRemoteStorageWith exposes the cache to other builders.
- BuildGuardedRemoteStorageClient caches the guarded client per
configuration. The endpoint is checked against the deny-list when a
client is built; every new connection still dials through the guarded
dialer. A gcs credentials file, and for an S3-compatible remote using
the default credential chain the shared credentials file and AWS_*
variables, are part of the key by content.
- The shared transport keeps 32 idle connections per host, caps
connections per host (WEED_REMOTE_MAX_CONNS_PER_HOST, default 64) and
has dial, TLS handshake and response header deadlines, so a hung
remote frees its slots.
- The filer drops the clients of removed remotes (and all but the newest
client per name) on every /etc/remote reload, also when /etc/remote
does not exist; reloads are serialized.
* remote: drop clients whose build spans a reload or whose remote is gone
* server: hash the shared AWS config file into the guarded credential version
* server: bound a stalled body on the shared remote transport
* remote: waive path-injection on the configured credentials file
* remote: keep caching remotes the local filer does not name
* remote: drop the inline codeql waiver the analyzer ignores
* remote: version the plain gcs client's credentials file
* remote: hash the environment's inline gcs credentials
GOOGLE_APPLICATION_CREDENTIALS can hold the JSON itself, which the gcs
client accepts; it is not in the configuration hash, so a changed value
kept the old client.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* remote: version the well-known gcs credentials file
With neither the configuration nor GOOGLE_APPLICATION_CREDENTIALS set,
the SDK reads application_default_credentials.json; rotating it left the
cache key unchanged and the old client behind.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* remote: share the gcs credentials version with the guarded client
The guarded gcs client resolves the same environment and well-known
credentials as the plain one; keying it only on a configured path left
those rotations invisible to the cache.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* remote: version s3 keys resolved from the environment
A remote whose access and secret keys both arrive through its type's
environment fallback (aliyun, baidu, contabo, filebase, storj, tencent)
never entered the default-chain branch of guardedCredentialsVersion, so
an in-process environment change kept serving the stale cached client.
The version now hashes the resolved keys for every S3-compatible remote.
* volume: reuse one remote storage client per configuration (rust)
fetch_and_write_needle built a fresh aws_sdk_s3::Client per request,
so every cold read kept its connector's idle connections without ever
reusing them — the same leak the Go side fixes. Cache clients keyed by
the configuration's protobuf encoding in an LRU of
WEED_REMOTE_CLIENT_CACHE_MAX entries (default 64), mirroring
weed/remote_storage/client_cache.go. An evicted entry drops the client's
last Arc and with it the connector's idle connections. The per-request
endpoint validation is unchanged: Rust has no dial-time guard yet, so
the up-front DNS check stays on every call.
---------
Co-authored-by: user.email <chrislusf@users.noreply.github.com>
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* mount: rebuild dropped directory listings on lookup
Invalidation, idle eviction, and a kernel Forget all clear a directory's
cached-listing flag, but only an expired listing was rebuilt on lookup — a
dropped one read through per entry until a ReadDir happened to run. Track
directories that once held a listing in wasCachedDirs, which survives those
drops, and let ShouldRebuildExpiredDir rebuild them on the next lookup.
Never-cached and read-through directories keep reading through.
* mount: re-validate a stale directory section on lookup
A burst of remote changes invalidates only the directory section it lands
in, but nothing re-validates that section until a ReadDir runs — so a
lookup-heavy workload reads through per entry indefinitely. EnsureNameFresh
re-lists just the section covering the looked-up name, after which lookups
in it are served from the cache again.
* mount: wait for pending flush before a stale-section refresh on lookup
A section refresh lists the range from the filer; without the wait a
pending async flush of the looked-up entry could land after the listing
and leave the refreshed section holding pre-flush metadata — the same
hazard the rebuild path already waits out.
* mount: bound the dropped-listing rebuild history
wasCachedDirs grew one full path per directory ever cached, removed only
by the read-through mark, so deleted and forgotten directories
accumulated forever. Remove the record when its name is deleted, move it
on rename so rebuild eligibility travels with the directory, and cap the
history with FIFO eviction — past the bound an evicted directory reads
through per entry, the same end state as a never-cached one.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* mount: drop oversized stale sections and back off failed refreshes
A section that outgrew one refresh failed EnsureNameFresh with
ErrRefreshRangeTooLarge on every lookup, repeating the multi-page
listing before each per-entry read, and other refresh failures retried
with no cooldown. Drop the cached listing on ErrRefreshRangeTooLarge so
a later lookup rebuilds it re-tiled, as ReadDir does, and stamp the
rebuild attempt on other failures so retries back off under the same
cooldown the rebuild gate uses.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* mount: version hard-link sibling syncs at the committed log position
* mount: tag dropped-listing queue slots so stale ones cannot evict
RemovePath, MovePath, and MarkDirectoryReadThrough delete the rebuild
record but leave the FIFO slot behind, so a path removed and re-cached
holds two queue entries. When the stale slot reaches the front it evicts
the newer record, and the directory reads through per lookup until the
next listing. Tag each record with an insertion generation and only
evict on a match, keeping the newer record for its own slot's turn.
* mount: refresh hard-link siblings from the filer, not the cached copy
Stamping a cached sibling with the mutation's version claimed that
position for content that may predate it — an in-flight event with
newer chunks or attributes then hits the version gate and the sibling
serves stale data until a refresh. Fetch each cached sibling's entry
from the filer after the commit and record the higher of the fetched
and committed positions, falling back to the in-place counter patch
only when the fetch itself fails.
* mount: back off failed section refreshes per section, not per directory
A failed refresh of one stale section marked a directory-wide attempt,
so every other stale section in that directory waited out the cooldown
with per-entry filer reads. Key the attempt timestamp by the section's
lower bound so unrelated ranges still refresh on their own schedule.
* mount: fence hard-linked entries by the group's newest write position
Every path of a hard-link group reads one shared content blob, but the
version gate only tracks per-path records — a sibling absent from
inodeToPath got no version from the sibling sync, so a stale section
refresh or event could roll the shared blob back for all of them.
Record a monotonic version floor per hard-link id wherever versioned
content lands, and check it in the section-refresh and event gates.
* mount: keep hard-link names landing when the group floor fences content
The per-group version floor rejected a delayed event's newEntry while
its oldPath still applied — a rename could drop the source without
adding the destination, and a section refresh could skip then mark a
live link fresh. Rebase the group's current cached content onto the
new path so the name lands without rolling the shared blob back, only
skip refresh entries already cached, and gate the sibling refresh
write under the same lock so a racing event or delete wins.
* mount: rebase hard-link entries from the shared blob when no sibling is cached
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* mount: park group content on uncached hard-link names a stale refresh installs
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* mount: stale the destination section when a fenced hard-link name cannot land
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* mount: bound the sibling hard-link refresh under one request deadline
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* mount: replace a reused-name occupant the hard-link floor would skip
* mount: claim no version for a sibling counter patched without a fetch
* mount: vacate a reused-name occupant without decrementing its group
A sibling write can fold the occupant's filer-side unlink into the shared
blob counter while the stale entry is still cached, so replacing it through
the plain insert may decrement a group that no longer counts the name and
drop a blob live siblings still read. Vacate the path like a rename first;
the counter then only ever drifts high, which leaks a blob rather than
losing shared content.
* mount: keep a fetch-failed sibling's version when patching its counter
The fallback marked the entry unversioned, which fences nothing — a
replayed delete older than the cached content could remove a live link.
Claim the version the entry already holds instead: newer in-flight events
still land and correct it, older replays stay fenced.
---------
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* admin: warn on dashboard when cluster nodes run mixed versions
A partially-finished upgrade leaves servers on different releases, and
nothing surfaces that drift today. The dashboard now compares the
binary version reported by every node:
- volume servers are asked directly via VolumeServerStatus (the
heartbeat topology carries no version), fanned out with bounded
concurrency and a 60s cache since versions only change on restart
- filers, S3 servers, and message brokers already report their version
over KeepConnected; ListClusterNodes returns it, the dashboard just
never read the field
- masters are excluded: raft membership does not carry versions
When every node matches, the Cluster panel shows the shared version.
When they differ, a warning banner calls out the mixed cluster and the
panel lists each node type with its count grouped by version, e.g.
"31x 4.39, 24x 4.44". Version columns on the volume server, filer, and
S3 tables (and the dedicated volume servers page) identify which nodes
are stale. Versions are compared without the volume-size-limit prefix
("8000GB 4.44 abc123" -> "4.44 abc123") so capacity config does not
masquerade as version drift; nodes reporting nothing count as
"unknown".
* admin: run the volume version queries concurrently, not under the map lock
The per-server VolumeServerStatus call ran while holding the versions
mutex, so the semaphore-bounded fan-out was fully serialized: every
server's RPC waited for the previous one, up to the 3s timeout each.
* admin: export the volume server table's real columns
The new Version cell shifted every later column, so the CSV read the
version as Rack and dropped Max Volumes, EC Shards and Usage a slot
early. Map the cells the table actually renders and add Version to the
export.
* admin: warn on mixed versions only when nodes report versions
An empty or master-only cluster produced a non-uniform summary with an
empty node list, which rendered the "Mixed cluster versions detected"
banner and warning icon over nothing. Mixed() requires reported nodes,
and the version card shows a dash when none answered.
* iceberg maintenance: bound decimal, time, uuid, fixed and rescaled timestamp columns on compacted files
The column statistics written for a compacted data file give lower and upper
bounds only for booleans, integers, floats, dates, strings, binaries and
timestamps already stored in the Iceberg type's unit. Every other column is
left unbounded, so a reader can never skip a compacted file on it: a
DECIMAL(18,2) column written by DuckDB (stored as INT64) loses its bounds
after compaction, as does a millisecond timestamp, a TIME or a UUID.
boundLiteral now also converts:
- DECIMAL stored as INT32 or INT64 at the column's scale. A decimal stored
as a byte array still gets no bound: Parquet compares those statistics as
bytes, which does not order negative two's-complement values;
- TIME, TIMESTAMP, TIMESTAMPTZ and TIMESTAMP_NS from any Parquet time unit,
rescaled to the Iceberg unit; a value rescaled to a coarser unit is
rounded down for a lower bound and up for an upper bound so it stays a
bound, and a value that would overflow gets none;
- UUID and FIXED[n] from a FIXED_LEN_BYTE_ARRAY of the right length;
- a DOUBLE column read from a FLOAT file.
arrow-go becomes a direct dependency (decimal128, already in the module
graph through iceberg-go).
* iceberg maintenance: bound TIME(MILLIS) columns stored as INT32
* iceberg maintenance: bound timestamptz_ns columns too
---------
Co-authored-by: user.email <chrislusf@users.noreply.github.com>
* iceberg maintenance: an unreadable parquet file fails the job, not the process
parquet.NewReader panics on any file it cannot open (wrong magic, a
truncated footer, zero bytes), and parquet-go's row decoder panics on a
corrupt page under a valid footer. Maintenance runs in the worker's
goroutines with no recover, so one bad delete or data file in any table
ended the whole process.
- Every parquet open in compaction goes through newParquetReader, which
opens with parquet.OpenFile and returns the error.
- A panic while decoding a delete file or a data file's rows is recovered
into an error naming the file (recoverRead/guardRead).
- As a backstop, detection recovers per table and Execute per operation:
the table is skipped or the operation fails, and the worker keeps
running.
A failed delete file still fails the table's compaction (no commit), as a
read error did before; a failed data file still skips only its bin.
* iceberg maintenance: test a corrupt page inside a merge fails it naming the file
* iceberg maintenance: assert a skipped bin's files stay in the snapshot
---------
Co-authored-by: user.email <chrislusf@users.noreply.github.com>
* iceberg maintenance: apply equality deletes only to older data files in their partition
Compaction applied every equality-delete key to every data file of every
bin. The Iceberg spec ("Scan Planning") applies an equality delete file only
to data files with a strictly lower data sequence number, and only to data
files in the delete file's own partition unless its spec is unpartitioned.
An upsert writer (Flink, the Iceberg Kafka Connect sink, CDC tools) commits
an update as the row's new version plus an equality delete of its key in the
same commit, so the compaction deleted both the old and the new version of
every updated row.
- equalityDeleteGroup is keyed by equality field ids and scope (unpartitioned,
or spec id plus partition) and keeps, per key, the highest data sequence
number of the delete files carrying it. A delete file without a data
sequence number is an error rather than a guess.
- Partition values are compared typed through the spec's partition type, so
one partition decoded in different Go shapes (a uuid as uuid.UUID or
[16]byte, a long as int32 or int64, a null present or missing) is one
partition. A spec whose fields are all void is unpartitioned. A delete whose
scope cannot be decided (unknown spec, no partition values under a
partitioned spec, a value that does not convert) stops the compaction; a bin
whose own partition cannot be typed while a delete of its spec exists is
not compacted.
- Each bin is merged with only the groups that apply to its partition, and a
row is deleted only by a key whose sequence number is higher than its data
file's.
- A where filter no longer drops unpartitioned delete files (the clause's
field is not in their spec), which let deleted rows return in a filtered
compaction.
Tests: the table fixture now commits its delete files after the data (data
sequence number 1, deletes 2), as writers do; hand-built entries in the unit
tests get sequence numbers. New eqdelete_sequence_test.go covers rows written
with and after their delete, partition scope, unpartitioned and all-void
specs, typed partition matching, missing sequence numbers and partition
values, where filters, and group-key collisions.
* iceberg maintenance: split the undecidable-scope branch in deleteInPredicateScope
---------
Co-authored-by: user.email <chrislusf@users.noreply.github.com>
* s3: accept bare composite checksum on CompleteMultipartUpload
Clients such as minio-go send the composite object checksum without the
"-<parts>" suffix. Compare against the base64 part of the computed
composite value as well as the full form.
* s3: test object checksum matching with and without part suffix
* filer.redis3: prefix every key via getKey, honoring keyPrefix
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* filer.redis3: read username and keyPrefix options into client config
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* filer.redis3: reject a username configured without a password
go-redis only sends AUTH when a password is set, so a configured username
with an empty password silently connects as the default user instead of
the intended ACL user. Fail loudly at initialize instead.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* filer.redis3: cover keyPrefix scoping in the store tests
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* filer.redis3: derive the test prefix and kv key from the per-run directory
---------
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* filer.remote.sync: skip an event the filer has written past, not only rewritten
isSuperseded only looked for chunks the event named that the filer no
longer holds. A file copied in through a mount logs a create while the
copy is still running: the mount flushes metadata with chunk uploads in
flight, so that chunk list can have holes. ChunkStreamReader stops at
the first hole, the multipart upload sends a short part ("ContentLength=
8388608 with Body length 4194304"), and since every chunk the event
names is still live the event was never superseded. The daemon retried
it forever and the offset never moved past it.
Treat chunks the filer has added since the event the same way. The
event that added them follows in the log and uploads the finished file.
* filer.remote.sync: skip a holed snapshot even while the filer still holds it
A mid-write snapshot the writer never finished is neither deleted nor
written past: the filer holds it unchanged, so the superseded check never
fires and every retry reads short at the same gap, pinning the sync offset
forever. The upload can never succeed for that entry — skip it like a
superseded one. A finishing write logs its own event and uploads the full
content; a dead write has nothing readable to lose.
The check also covers a chunkless entry that is not a dropped local copy
(e.g. a truncated remote-only entry): its reader yields zero bytes against
a nonzero size, which can silently clobber the remote object with an empty
body where the backend does not verify the length.
* filer.remote.sync: upload a holed entry zero-filled instead of skipping it
Skipping the event left a completed sparse file absent or stale on the
remote, and the coverage check on the raw chunk list missed both gaps a
manifest's bounding extent hides and a FileSize smaller than the chunk
extent. ChunkReadAt already resolves manifests the way the upload reader
does and zero-fills every uncovered range up to the effective size, so
route the upload through it: a sparse file, a mid-write snapshot, and a
chunkless entry all send the bytes a read of the entry returns, and a
hole can neither wedge the event nor publish a truncated object.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* filer.remote.sync: skip a superseded entry on reader-construction error too
---------
Co-authored-by: Alex Russell <alex@sengine.cloud>
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* s3: return x-amz-copy-source-version-id from CopyObject and UploadPartCopy
* s3: cover x-amz-copy-source-version-id on latest copies and part copies
The explicit-version case was already covered; add a .versions pointer
fixture so the unversioned request resolves the latest version and
asserts the header on CopyObject, plus the same assertion on
UploadPartCopy.
* s3: carry the resolved id on a version file that lacks it
A .versions child written before the version-id attribute resolves to an
empty source version even though the pointer or the rescan already knows
it. Stamp the id on the returned entry so copy and get readers report it;
the version copy tests now drop the attribute to cover that shape.
* s3: report the null version in x-amz-copy-source-version-id
A regular-path source in a versioning-configured bucket is the null
version; echo it like every other version id, matching the GET path and
AWS. Unversioned sources still report no header.
* s3: report the version a self-copy retry actually read
---------
Co-authored-by: user.email <chrislusf@users.noreply.github.com>
* s3: allow PutBucketAcl when auth is off
* s3: resolve the PutBucketAcl owner against the recorded owner
The unauthenticated path now resolves the account as anonymous, but XML
ACLs were still checked against it while GetBucketAcl reported an empty
owner for ownerless buckets, so a read-modify-write ACL update failed
with AccessDenied. GetBucketAcl and the XML check now both use the
recorded owner when the bucket has one and the resolved account
otherwise. The canned-ACL grantee moves to the same owner, and a request
carrying no account no longer erases a stored owner.
* s3: cover PutBucketAcl without auth for XML bodies and stored owners
The XML case posts the anonymous owner GetBucketAcl reports for an
ownerless bucket; the canned case asserts a recorded owner survives an
unauthenticated update and receives the full-control grant. Ambient AWS
credentials are cleared so the memory store never turns auth on.
* s3: keep the recorded owner through a PutBucketAcl update
The grantee and the stored owner were resolved differently: grants
named the recorded owner while config.Owner still took the caller's
account, so an admin updating another account's bucket rewrote the
owner and broke an XML retry naming the original one. An ownerless
bucket adopts the ACL's owner; an owned bucket keeps it.
* s3: fall back to the admin owner for an ownerless bucket in ACL handlers
An unauthenticated PutBucketAcl resolved the owner as anonymous and stored
it on the bucket, but the registry reports an ownerless bucket as
admin-owned: BucketOwnerEnforced uploads then stamped anonymous instead of
admin, and ownership checks after enabling auth compared against the wrong
owner. Both ACL handlers now resolve an anonymous fallback to the admin
account, so GetBucketAcl reports the same effective owner the registry
does and the stored owner matches it.
* s3: resolve the admin owner for bucket grants when the caller is anonymous or untrusted
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* s3: skip recording anonymous identity or account as bucket owner
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* s3: report stored bucket ACL grants verbatim
Explicitly saved anonymous full-control grants must survive a
GET-to-PUT round trip unchanged, so the read path no longer rewrites
them to the reported owner.
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* s3: expect the stored anonymous grant verbatim in GetBucketAcl
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: user.email <chrislusf@users.noreply.github.com>
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
isSuperseded only looked for chunks the event named that the filer no
longer holds. A file copied in through a mount logs a create while the
copy is still running: the mount flushes metadata with chunk uploads in
flight, so that chunk list can have holes. ChunkStreamReader stops at
the first hole, the multipart upload sends a short part ("ContentLength=
8388608 with Body length 4194304"), and since every chunk the event
names is still live the event was never superseded. The daemon retried
it forever and the offset never moved past it.
Treat chunks the filer has added since the event the same way. The
event that added them follows in the log and uploads the finished file.
* s3: read regular copy sources from their write owner
* filer: report mutation sentinel codes in ObjectTransaction responses
A failed mutation only surfaced resp.Error text, so routed writers could
not distinguish an existing-directory conflict from a generic failure and
could not apply the prefix-object retry the plain CreateEntry path has.
* s3api: retry a routed mkfile as a prefix object over a directory
mkFile retries ErrExistingIsDirectory by marking the entry a prefix
object; the routed path returned the conflict instead, so a routed write
to a key that is also a prefix failed where the lock path succeeds. The
filer now reports the code on the transaction response, making the retry
recognizable.
* s3api: commit copy destinations on the key's write owner
An owner-routed source read still left the destination on the gateway's
default filer, so a second copy resolved the first copy's destination on
the owner, found nothing, and returned NoSuchKey. Write unversioned and
suspended destinations through routedMkFile, and write the version file
on the owner the finalize recomputes on.
* s3api: cache remote objects on the filer that served the entry
An owner-routed lookup can return a remote-only entry that has not
replicated to the gateway's default filer; the cache RPC then cannot
find it and the copy or read degrades to a 503. Ask the entry's write
owner to cache it first, falling back to the default filer.
* s3: read versioned object state from the key's write owner
CopyObject destinations now commit version files and the .versions
pointer on the owner's object-scope ring key, so versioned reads through
the default filer could miss them until replication. Generalize the
routed read to dir/name lookups and paginated lists scoped to the
object's owner, and point the versioned read paths at it:
getSpecificObjectVersion, doGetLatestObjectVersion, the .versions
quick-checks in Get/Head/attributes, the self-heal rescans, tagging,
delete conditionals, directory-marker history checks, rename
destination, multipart completion checks, and the version-id format
probe. Reads fall back to the default filer when no ring owner is
resolvable, preserving single-filer deployments.
* test(s3api): fake RECOMPUTE_LATEST/DELETE and cover versioned copy reads
The ambiguous-PUT fake filer now creates parent directories on PUT
(mirroring InsertEntry), applies DELETE mutations, and re-derives the
.versions pointer for RECOMPUTE_LATEST the way applyRecomputeLatest
does, so owner-vs-replica tests can exercise versioned writes.
TestVersionedCopyDestinationReadsOnOwner copies into a versioned bucket
whose writes commit on the owner while the gateway's default filer is a
replica with no replication, then asserts reads by versionId,
latest-version resolution, and a copy chained off the new version all
resolve on the owner.
* test(s3api): gofmt the versioned-copy coverage
* s3: only swap in a cache fill of the resolved remote generation
A cache response can come back from a filer holding an older generation
of the key. Replacing the authorized source or swapping its chunks under
a size and range computed for the resolved entry serves the wrong object
or truncates the response. Match RemoteEntry identity and size before
using a cached entry, in both the copy and streaming paths.
* filer: keep directory attributes when an object lands as prefix object
The object-onto-directory retry replaced the existing directory's mode,
owner, and extended attributes with the object's. Preserve the directory
identity; merge directory-held extended keys only when the old entry
never held an object, so a prior generation's object metadata is not
carried forward.
* s3: keep marker lock checks on the filer that replaces the marker
The object-lock precondition read the key's write owner while mkdir
still replaces the marker on the default filer, so a locked marker not
yet replicated to the owner could be overwritten. Read the entry where
the replacement lands, in both PutObjectHandler and zip extraction.
Also hash a trailing-slash marker key to the same route key as the
object it names, so marker and object share one owner and one write
lock.
* s3: route tag writes to the owner tag reads resolve
GetObjectTagging reads the key's write owner, but PutObjectTagging and
DeleteObjectTagging still wrote through the default filer, so a
successful change returned the owner's stale tags until replication.
setTags, rmTags, and the versioned tag updates now write on the owner
first, falling back to the default filer when no owner resolves.
* s3: keep rename lookups on the filer that performs the move
AtomicRenameEntry and the retry token it carries both execute on the
default filer, but the source and destination reads went to the write
owner — a completed rename's token was invisible to its own retry, and
a source that only exists on the default filer looked absent. Read
both ends on the default filer.
* s3: send the .versions pointer repair with the object's route key
The heal reads the .versions entry on the object's owner, but the
conditional UpdateEntry is routed by the .versions path's own ring key,
which can hash to a different filer — leaving the pointer the routed
reads consult unchanged. Ship the CAS repair as a PATCH_EXTENDED
ObjectTransaction on the object's route key instead.
* filer: preserve only directory identity on a prefix-object write
Merging the old directory's extended attributes could carry a marker's
encryption, version, or etag fields onto the new plaintext object, since
a directory's own keys cannot be told apart from the object it held.
Keep only mode, uid, gid, user name and group names.
* s3: also require matching size when a cache fill replaces the source
The remote-generation check covers identity; size equality is the cheap
guard that keeps the streamed or copied range consistent with what was
resolved.
* s3: lock the object path for a routed .versions pointer repair
conditionalUpdateEntryRouted held the lock on the .versions path while
version writes and deletes serialize on the object path, so a repair
could pass IF_ENTRY_EQUAL and then apply over a pointer a writer had
advanced in between. Lock the object path like the writers and keep the
condition on the .versions entry via condition_key.
* s3: do not replay an owner-side tag mutation on another filer
withWriteOwnerFilerClient re-ran fn on the default filer after any owner
error, so an authoritative not-found could update a stale replica entry
and still report success, and a post-apply transport error could apply
the mutation twice. The failover walk now ends on any error fn returns;
only a channel that never came up moves to another filer.
* filer: keep a directory's xattrs when a prefix object replaces it
The prefix-object branch preserves the directory's identity but dropped
its Extended map wholesale; FUSE xattrs live there under the xattr-
prefix and belong to the directory, not to the object the marker held.
* s3: delete null and specific versions on the filer that serves them
On the default filer an absent entry deletes as success, so a version
that only reached the write owner survived a reported success while
owner-routed reads kept serving it. The suspended-versioning and
lifecycle deletes now go through routedDeleteNullVersion and
routedDeleteSpecificVersion when an owner resolves; an unsettled
attempt keeps the lock path.
* s3: route a lifecycle null-version delete to the object entry
The "null" version is the regular object entry, not a .versions file,
so it goes through routedDeleteNullVersion; only a real version id takes
the .versions-file delete.
* s3: repoint .versions only while it still names the deleted version
routedDeleteSpecificVersion recomputed the latest pointer on every
version delete, so removing a noncurrent version re-derived the pointer
to the newest remaining file — resurrecting an older version over a
cleared pointer (a suspended bucket's null write keeps it clear while
the null object stays current) or over a pointer that moved on. The
pointer is now read on the owner and repointed under an IF_ENTRY_EQUAL
guard only while it still names the deleted version; a contested pointer
is left for the self-heal path rather than overwritten by a stale scan.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* s3: let lifecycle expiry keep its metadata-only deletes on routed paths
The routed lifecycle deletes hardcoded IsDeleteData, so expiring an entry
with a volume TTL queued per-chunk deletions instead of leaving the data
for the volume to reclaim — extra work on the owner for deletes
recordMetadataOnlyIf still counts as metadata-only. Pass the caller's
cleanup choice through routedDeleteNullVersion and
routedDeleteSpecificVersion; the interactive DELETE path keeps deleting
data.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* filer: let RECOMPUTE_LATEST skip a pointer that moved on
A version delete repoints .versions only while the pointer still names the
deleted version, but the gateway cannot carry the pointer's current state
into the transaction's condition — the condition slot is already spent on
the version's WORM guards. only_if_current_name moves the check under the
transaction lock: the filer compares the pointer's name_to_key value before
scanning and skips the recompute when it moved (or was cleared by a
suspended bucket's null write), so a noncurrent delete cannot resurrect an
older version.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* s3: repoint inside the version's locked delete
repointLatestBeforeVersionDelete ran in its own transaction ahead of the
delete, so a version delete denied by retention or legal hold still moved
the latest pointer — GET and HEAD went on serving the older version it
rolled to. The repoint and the delete now ride one transaction behind the
version's WORM guards, atomically gated by the new only_if_current_name
recompute guard instead of a stale-read IF_ENTRY_EQUAL.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* vector: Lease::forget hands the tag name off without releasing
A distributed planner pins the planned version and forwards the tag
name to the executor — the guard must not release on drop.
* vector: forget untracks the lease so an abandoned handoff expires
The tag left LIVE_TAGS but the drop-time release was disabled — an
abandoned handoff survived forever locally. Untrack it so the lease
TTL still reaps it.
* vector: DeleteIndex format guard, sticky discovery, advertise normalize
- handleDeleteIndex verifies the table carries the Lance format before
DeleteTable — an Iceberg table under the same namespace is not an
index and must not be removed by it.
- discovery keeps the current worker while it is still registered —
membership-order churn can no longer cancel in-flight calls, and the
resolver closes each master conn inside a 5s deadline.
- advertised_addr strips scheme and .grpcPort so the UDP probe reaches
every supported master form.
* vector: empty-follower retry, format check without read grant
- discoverVectorWorkers keeps looking past an empty master reply —
followers list nothing when workers registered with the leader.
- the DeleteIndex format check uses the internal catalog read, so
DeleteTable-only callers don't gain a GetTable requirement, and the
format must be Lance exactly — format-less tables no longer pass.
* ec: let shard-interval recovery exclude a set of suspect shards
gatherEcShardIntervals and recoverOneRemoteEcShardInterval skipped only
the shard being rebuilt. A needle-level rebuild wants every suspect
shard kept out of the gather so a bad copy cannot poison the
reconstruction it is being compared against.
* ec: rebuild a needle's suspect shard ranges when it fails to parse
A bit-rotted data shard serves a clean-looking but wrong interval, so the
missing/unreadable-shard fallback never engages and the needle CRC check
in ReadEcShardNeedle returns the corruption to the caller. Each interval's
source shard is a suspect: rebuild the suspect ranges from the surviving
shards — all of them at once while parity covers the suspect set, or one
at a time when the needle spans more shards than that — and let a
successful needle parse arbitrate which copy was corrupt. Shards whose
range changed are dropped from the location cache so later reads go
straight to reconstruction.
* ec: mirror the corrupt-shard needle rebuild in the Rust volume server
The snapshot now keeps every interval's shard placement so a parse
failure can name its suspects. rebuild_ec_shard_needle rebuilds the
suspect ranges through recover_one_remote_ec_shard_interval with the
suspect set excluded from the gather, and the winning candidate's
differing shards are dropped via forget_shard_location.
* ec: recover suspect ranges under the requested needle id
A failed ReadBytes re-reads the id out of the possibly corrupt header, so the rebuild passed whatever it decoded to peer recovery requests; carry the requested id through instead, and reject a candidate whose parsed id does not match it.
* ec: retry a failed needle rebuild with each gathered input dropped
The gather behind a suspect-range rebuild can itself serve a second corrupt shard: Reed-Solomon then reconstructs wrong bytes and the needle parse rejects the only candidate. Gather every surviving shard once, then re-decode with each of the first decode's inputs dropped in turn; the parse arbitrates the honest subset.
* ec: mirror the dropped-input rebuild and id check in the Rust volume server
Same two guards as the Go side: a candidate that parses under a different needle id is rejected, and a failed candidate whose gather fed it a corrupt input is retried by dropping one gathered shard at a time. The gather is shared through a want-bounded gather_ec_shard_intervals so the retry pays no extra fan-out.
* ec: answer deleted when a rebuild's holder reports the needle deleted
The rebuild discarded the deleted flag its interval recovery collected, so a needle deleted on its other holders but live in the local index came back as a corruption error. A deleted report is authoritative, so it now short-circuits the hypothesis search and answers ErrorDeleted.
* ec: hydrate rebuild candidates into a scratch needle
A failed ReadBytes leaves partially parsed fields on the needle, and optional sections do not reset when absent, so a corrupt first parse could leak stale name/mime/ttl into a successful candidate. Parse each candidate into a fresh needle and adopt it only on a matching id.
* ec: corrupt parity+1 shards in the exceeds-parity read test
The loop skipped the suspect's own shard when it fell inside the parity range, leaving only ParityShards corrupt shards — a case recovery can legitimately fix. Count the additional corruptions so the test always exercises the boundary it names.
* ec: mirror the deleted-report short-circuit in the Rust volume server
A holder's deleted answer during a rebuild propagates out as EcMiss::Deleted instead of surfacing as InvalidData, matching the Go path's authoritative-delete rule.
* master: replicate the recovery-wait growth unfreeze through raft
state_loss_recovery_wait lifted the volume-growth freeze only in
memory, while the freeze itself was replicated. A restart replayed the
frozen flag and growth stayed refused until the wait expired again, on
every restart, unless an operator ran volume.growth.resume.
The lift now applies GrowthFrozen=false like ResumeVolumeGrowth, from a
goroutine since IsGrowthFrozen is a read path that cannot block on
raft. The apply runs under reportedTopologyIdsLock so a freeze landing
after the lift keeps its command last in the log; the stale unfreeze
skips itself when the flag is frozen again.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* master: cover raft replication of the recovery-wait unfreeze
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* master: retry and re-arm a growth unfreeze raft failed to record
applyGrowthFreeze swallowed the raft error, so a rejected lift during a
leader change left the flag clear while the log still held the freeze;
nothing retried and a restart re-froze growth anyway. The apply now
returns the error, replicateGrowthUnfreeze retries it, and on
persistent failure re-arms the flag so the next IsGrowthFrozen
attempts the lift again instead of diverging from the log.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* master: release the topology-id lock before the freeze raft write
CheckReportedTopologyId and SetTopologyInitPending held
reportedTopologyIdsLock across the blocking raft apply, so a slow
apply stalled every later report — including the heartbeat that
should trigger the next freeze. The lock now covers only the
mismatch decision; the replicate step runs after it is released.
* master: bind the recovery unfreeze to the freeze instant it lifts
An unfreeze submitted while a newer freeze sat committed but not yet
applied — e.g. after this node won an election — could land behind
that freeze in the log and erase it. The command now carries the
freeze instant cluster-wide (GrowthFrozenAt) and the lift is bound
to the instant its submitter observed (GrowthUnfreezeFor): the state
machine no-ops a lift while a strictly newer freeze exists, which
also lets the unfreeze run without holding the topology-id lock at
all. Concurrent freeze writes keep their own mutex so log order
cannot regress the instant.
* master: a superseded unfreeze retry leaves the resumed flag alone
replicateGrowthUnfreeze broke out of its retry loop when a newer\nfreeze superseded the lift, but the break fell through to the\nre-arm: an operator resume in between was silently undone. Return\ninstead — the newer freeze generation owns the flag.
* master: replayed freeze entries without an instant stay ageless
Freeze commands written before the instant field existed carry zero.\nSubstituting the replay time made each restart stamp the freeze as\nnew, so a committed bound unfreeze always looked older and was\nrejected — the freeze came back on every restart. Keep the zero\ninstant: the recovery window reads as elapsed and bound lifts apply.
* master: pack the growth freeze flag and generation into one atomic
Separate flag and timestamp atomics left three holes: a resume's\nload-and-clear could lift a generation stamped in between, a bound\nlift compared stamps numerically although masters' clocks need not\nbe monotonic, and an ageless generation had no identity a bound lift\ncould match without colliding with the unconditional encoding.\n\nOne value now carries the state — 0 unfrozen, -1 an ageless freeze,\nelse the freeze instant — so every transition is a single atomic\nswap or compare-and-swap bound to an exact generation.
* master: a repeat mismatch report keeps the armed freeze generation
Re-stamping while already frozen never reaches the raft log, so the local generation diverges from the replicated one and every bound unfreeze becomes a no-op on replay — the freeze returns after restart or failover.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* master: snapshots read the freeze flag and generation in one load
Save, Snapshot, and PersistMaxVolumeId read IsGrowthFrozen and the generation separately, so a resume landing between them persisted frozen=true with stamp 0 — restoring an ageless freeze a committed bound lift cannot clear.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* iceberg: serve a registered table's metadata on load and commit
RegisterTable saved only the metadata location, and loads and commits
never read the file there. They served a placeholder with no schema, no
snapshots and a new random table UUID on every load, so a registered
table read as empty, an INSERT failed against the empty schema, and
every other commit failed its assert-table-uuid requirement. Metadata
uploaded through the S3 gateway, as a migrated table's is, is stored in
chunks that registration did not read, so it registered an empty table.
Read the file following its chunks, reject one that is not Iceberg
metadata with a 400, and keep its content and table UUID in the record
the way CreateTable does.
Records registered before this change hold only the location. Loads and
commits now read the file there, and the first commit stores it in the
record under the file's table UUID.
* iceberg: transactions read a location-only registration's metadata file
* iceberg: cover transactions on a location-only registration
* iceberg: refresh the default name mapping on transaction commits
* iceberg: cover name-mapping refresh in transactions
* iceberg: roll a transaction back to a registration's metadata file
A transaction that failed after flipping a table registered before the
record kept its metadata could not undo the flip. The rollback restored
the record's prior metadata, which for such a table is only a location,
and UpdateTable keeps the full metadata a flip stored when the restore
sends none. Loads went on serving the change the client was told had
failed, and a client retrying the transaction could apply it twice.
Restore the metadata loads served from the file instead, under the
file's table UUID.
* iceberg: register oversized metadata by its location
Registration sends the metadata file's content to the S3 Tables
manager, which refuses a request over its 10MB body limit, so a table
whose metadata is larger could no longer be registered at all.
Register such a file by its location and table UUID alone; loads read
it from that location, as they do for a record that never kept its
metadata. Export the limit as s3tables.MaxRequestBodySize so the
catalog can check against it. A commit to such a table still sends the
whole metadata to UpdateTable and stays over the limit.
* iceberg: commit oversized metadata by its location
* iceberg: let maintenance follow a location-only table record
* admin: read a table's metadata file when the record keeps none
* iceberg: bound the metadata file a location-only record reads
* iceberg: bound only chunked metadata reads in the worker
Inline content is what the catalog itself wrote and the REST load path
accepts it at any size, so rejecting it in loadMetadata stranded tables
over 64 MiB from every maintenance scan. Cap only the chunked read, where
the recorded location can name any object in the bucket.
---------
Co-authored-by: chrislusf <chrislusf@users.noreply.github.com>
* sftp: fail rm and rmdir of a missing path, and mkdir of an existing one
* sftp: create directories exclusively
---------
Co-authored-by: user.email <chrislusf@users.noreply.github.com>
* topology: add AvailableSpaceWithinLocation for location-scoped free slots
AvailableSpaceFor answers for the whole topology; a location-constrained
assign needs to know whether the requested data center, rack, or data node
still has room before deciding growth can satisfy it.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* master: let location-constrained assigns wait for pending growth
An assign carrying a data center or rack constraint failed on the first
unsatisfied pick, even when the same request had just queued volume growth
for that location — so the first write into a fresh collection errored out
instead of picking up the volume growth landed a moment later.
Keep the fail-fast path only when growth cannot help: growth disabled, no
growth pending, or no free slots left inside the constraint. Otherwise the
request takes the same wait/shed path as an unconstrained assign.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* topology: sum leaf free slots in AvailableSpaceWithinLocation
Rolled-up rack and data center counters keep a maintenance-mode server's
capacity, and one overcommitted node can cancel a sibling's slots. Sum the
matching data nodes directly so the location check agrees with what growth
can actually use.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* master: let constrained growth initiators reach the shed path
When a constrained assign's own growth concluded unfulfilled, the fast-fail
check saw only that no grow was pending and returned the plain pick error —
a code the assign client does not retry, so the first write died instead of
re-triggering growth. Exclude the request's own initiated grow from the
no-pending-growth condition so it reaches the existing re-pick and
ResourceExhausted shed like an unconstrained assign.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* master: judge targeted grow requests by the location they name
A grow request carrying a data center, rack, or data node constraint was
discarded whenever the layout's writables elsewhere were healthy, so a
location with free slots but no writable could never receive the volume its
assign was waiting for. Add ShouldGrowVolumesWithinLocation and use it for
constrained requests; the layout-wide check still governs untargeted ones.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* topology: only positive leaf slots count in AvailableSpaceWithinLocation
An over-committed node reports negative free slots; summed plainly it\ncancelled a sibling's room and a constrained assign could fail fast\nwhile the location still had space to grow into.
* master: fail constrained assigns fast when growth cannot place
A constrained assign with free slots in its location still waited out\ngrowth that could never satisfy the replica placement — a single-rack\ndata center asked for 010 sheds the same doomed grow on every client\nretry. Run the grow-time node search as a dry run and return its\nplacement error instead of a retryable shed.
* master: judge constrained growth by layout, not free slots
A slot-level feasibility probe races with the grow it judges: a\npending grow consuming the last slot in one rack reports the\nplacement impossible while it mints the request's own volume, and\nthe hard error skips the re-pick that would find it. GrowthBlocker\nnow counts racks and data nodes inside the named location — shape a\ngrow cannot consume — so only genuinely unplaceable layouts fail\nfast.
* topology: GrowthBlocker mirrors the grow selector's feasibility
The location filter bound every replica tier, so a 010 request naming one rack needed two racks inside it while the selector constrains only the first placement and counts the rest by room. And counting nodes without free slots let a full location queue a grow it could never serve, so the assign shed ResourceExhausted at timeout instead of failing fast. GrowthBlocker now applies the option's location to the primary placement chain only and leaf-sums free slots the way the selector does, so it reports a blocker exactly when the grow can never place.
* master: repick once before surfacing a growth blocker
An in-flight grow can spend the last free slot minting the request's own volume: the blocker's slot counts predate that registration by a heartbeat, so returning the placement error on the first failed pick could reject a write whose volume lands a moment later. A pending grow now earns one repick grace before the blocker error is returned.
---------
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* master: end the post-mint allocation hold on the embedded reporter's clean inventory
DeferAllocation armed a warmup-length hold whenever the leader minted a
fresh topology id, and on weed server the hold ran the whole window: the
embedded volume server's join kept refreshing the quiet evidence clock, so
the two-pulse early release still cost the first write 10-15s on every
fresh start. weed mini already releases on its single pinned reporter's
first inventory, but pinning refuses outside volume servers, which weed
server must keep admitting.
Add a non-exclusive variant: the topology names one expected reporter
whose first clean full inventory lifts the hold, while other reporters
stay admitted and still feed prior-state evidence and the topology-id
freeze. A single-master weed server points it at its embedded volume
server — the one reporter guaranteed to arrive quickly; a veteran's
heartbeat carries prior state and keeps the window through the regular
checks, so a wiped -dir with surviving external volume servers still
trips the freeze on first report. Restarted clusters are untouched: a
recovered topology id means no mint and no hold.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* filer: announce on the startup peer scan for embedded single-master setups
A fresh filer whose startup peer scan came back empty re-polled for the
full confirm window before registering on the master, so the lock ring
had no member and routed traffic such as S3 PUTs spun on "no lock server
found" for ~10s on every fresh weed server / weed mini start.
For setups that start with exactly one filer the scan is already final:
a single-master weed server and weed mini announce as soon as it
completes. A late external filer still joins and replays through the
normal path — the confirm window only ever caught peers arriving inside
it, and the residual risk mirrors the accepted volume-side trade-off: an
embedded filer announcing a few seconds earlier can own the lock ring
while a surviving external filer's backlog is still on its way.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* volume: retry a failed heartbeat promptly until first registration
A heartbeat stream refused before the leader is known — the common race
when a volume server starts while an election is still running — always
fell back to sleeping out the full pulse, so a fresh volume's first
registration could wait ~5s behind a leader elected a second later.
The peer refresh already identifies the leader during the wait, so cap
the retry pause at a second once a leader is named; while no leader is
known the same short retry applies until the first stream is confirmed,
since an unregistered server has nothing else to do. Once the server was
registered the pulse backoff is unchanged, so a lost leader still paces
reconnects the way it did before.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* volume: retry a failed heartbeat promptly until first registration (rust)
Mirror the Go heartbeat retry: cap the post-refresh wait at a second when
a leader is named, and keep the short retry until the first stream is
confirmed so a fresh volume registers promptly when it races an election.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* master: leave admitted reporters a confirm pulse when the embedded report ends the hold
The single-master weed server releases its post-mint allocation hold on
the embedded volume server's clean report, but outside volume servers
stay admitted and one may still carry prior cluster state — an inventory
from the embedded server alone cannot prove none is due. Clearing the
hold outright let a rebuilt master re-issue a volume id a reconnecting
veteran still owned.
Collapse the released hold to one heartbeat interval instead: late
reporters land their evidence or a topology-id freeze inside the pulse,
and a re-mint re-arms that pulse rather than the full warmup. The pinned
single-reporter release (weed mini) still drops the hold outright, since
no other reporter can be admitted.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* filer: keep the empty-membership confirm on embedded single-master weed server
A single-master weed server still admits external filers, so an empty
startup peer scan is not final: after a master restart the existing
filers simply have not reconnected yet. Announcing on the scan let the
embedded filer take lock-ring ownership and serve writes that a peer's
replayed backlog could then overwrite.
Only weed mini keeps the skip, where the embedded filer is the only
member its dev cluster can have. The confirm window only delays the
master listing for a fresh store; the filer serves while it runs.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* volume: tolerate an expired heartbeat retry deadline (rust)
refresh_leader_from_peers can consume the whole pulse budget, leaving
retry_at in the past; subtracting a later Instant panics and kills the
heartbeat task instead of retrying. Match the Go side, where
time.Until of a past deadline just sleeps zero.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* volume: retry the seed list without a second pulse sleep (rust)
The error path already slept its retry delay; an extra full pulse after
losing the round delayed the next attempt past what Go's heartbeat()
does, which paces retries only inside the per-master attempt.
* master: keep a configured allocation hold shorter than the confirm pulse
Once the embedded report lands, the collapsed hold replaced the
configured deferral with a full pulse even when the operator set a
shorter one, including zero (disabled). Cap at the pulse instead.
* filer: end the empty-membership confirm early on a fresh-minted embedded master
On a fresh single-master weed server the announce gate still held the
filer for the full confirm window, so the master's lock ring stayed
empty and the first write retried 'no lock server found' for ~7s on
top of the collapsed allocation pulse. A fresh TopologyId mint proves
the master's persisted state was empty, making the empty peer scan
final; skip the rest of the confirm only then, and keep it whenever
the master recovers a prior id.
* filer: keep a short reconnect window on a fresh-minted embedded master
A fresh TopologyId mint proves the master's persisted state was empty,
not that no external filer survived its loss — one can still be
mid-reconnect on its keepalive cycle when the confirm starts. Ending
the empty-scan confirm on the first poll let the fresh filer register
and serve routed reads before that peer was found and backfilled, so
existing files could read as missing. The fresh-mint path now shortens
the confirm to a few keepalive cycles instead of skipping it: a
reconnecting filer still lands in time, while a genuinely fresh cluster
announces after ~3s rather than ~10s.
* volume: skip the heartbeat loop when no seed masters are configured
An empty seed list made the outer loop spin with no await between
passes and no path to observe the stop flag. A masterless volume server
has nothing to register with, so the heartbeat exits after a warning.
* volume: skip the heartbeat loop when no master addresses are given (rust)
Mirrors the Go heartbeat: an empty seed list spun the outer loop with
no await and no shutdown checks. Exit after a warning so library
callers get the same behavior as the binary path that validates the
list up front.
* filer: re-check the fresh-mint hint on each confirm pass
The topology id is minted after raft election, which can land after
the empty-scan confirm has already started — a one-time check at entry
missed the mint and held the full window. Evaluate the hint on every
poll and shorten the confirm from the first observed mint, still
capped by the normal confirm deadline.
* master: mark a fresh topology mint only when the volume bound is zero
Recovery restores the max volume id unconditionally but the topology id
only when present, so an upgraded or partially rebuilt master can mint a
new id while still holding prior volume state. Marking fresh there would
let the embedded filer shortcut its peer confirm on a cluster that is
not empty; only mark the mint fresh when no bound survived.
* filer: drop the fresh-mint shortcut once a peer has been observed
---------
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* s3: allow PutObjectAcl when auth is off
* s3: ownerless objects claim the admin account when auth is off
An object written through the filer carries no owner metadata, so
objectOwner fell back to an empty account id and ExtractAcl rejected the
canned ACL's canonical grantee. Extend TestPutObjectAclWithoutAuth to
cover the ownerless case.
* s3: GetObjectAcl reports the admin owner for ownerless objects when auth is off
The same empty-owner fallback produced an ACL response with a blank
Owner ID and a blank canonical grantee.
* s3: ignore the client-supplied account id when auth is off
Auth does not strip AmzAccountId on the iam-disabled path, so the
ownerless fallback could persist a forged owner. requestAccountId
substitutes the admin account, matching the idiom already used in the
upload ACL path.
---------
Co-authored-by: user.email <chrislusf@users.noreply.github.com>
* vector: only follow a leader hint that points elsewhere
The master sends volume_location on every KeepConnected response —
the leader always names itself, so the worker exited the stream on
the first response and flapped every 3s, leaving ListClusterNodes to
see the node intermittently. Only redirect when the hint points at a
different master.
* vector: leader hints carry host:http.grpc — normalize before comparing
A self-hint arrives as host:9333.19333 while --masters gives host:9333,
so the raw comparison still redirected and grpc_addr could not parse
the dotted port. grpc_addr now prefers the explicit .grpcPort, and the
hint comparison normalizes both sides to host:httpPort first.
* vector: owner-prefixed lease tags; foreign leases get a long backstop
reap_expired exempted only process-local LIVE_TAGS — server B could
reap server A's live query lease at the 1h TTL. Tags now carry a
per-process instance id: own leases use the short TTL, foreign leases
only the 24h backstop, so a live query elsewhere keeps its pin while
dead-process tags still collect eventually.
* vector: pick the lease-tag format by segment count, not numeric-ness
An owner that is all decimal digits parsed as the legacy timestamp, so
its real timestamp was ignored and the lease never expired. The third
segment is what distinguishes {instance}-{ts}-{uuid} from {ts}-{uuid}.
vector: cover the index name in named_segment_fragments
The planner groups tasks by index name — assert the helper actually
returns the logical name so a regression dropping it fails.
vector: cover the identity object in the s3vectors auth context
The auth middleware must carry the resolved identity object — only
the name survives otherwise and authorization loses actions, account,
and claims.
* sftp: refuse mkdir and uploads under a missing folder
* sftp: re-check the parent directory when a buffered upload commits
The open-time check cannot catch a parent removed while the upload
buffered; the filer would recreate it when the entry lands.
* sftp: cover mkdir and upload commit against a missing parent
Extend the test filer with CreateEntry so makeDir can be exercised end
to end.
* sftp: refuse a rename into a missing folder
The filer's rename path creates the target entry through CreateEntry,
which recreates absent parents the same way uploads did.
---------
Co-authored-by: user.email <chrislusf@users.noreply.github.com>
* s3vectors: cover non-filterable key round-trip and nested filter rejection
The catalog writes the key list as a JSON string while older records
may still carry the array form; indexFromRecord must decode both. The
rejection path extracts field names through nested $and/$or filters, so
a blocked key deep in the tree still fails the query.
* s3vectors: factor the non-filterable rejection into a testable gate
Pull the blocked-key check out of handleQueryVectors into
nonFilterableField so the rejection itself is covered without standing
up the catalog stack; the handler now 400s on whatever the gate names.
* s3vectors: report the non-filterable match separately from the key
An empty metadata key in nonFilterableMetadataKeys collided with the
"no match" sentinel, so a filter on that key skipped rejection. Return
a match flag and cover the empty-key case in the test.
* wdclient: cover a lapsed caller deadline that Err() has not observed yet
* wdclient: reuse isCanceledErr for the elapsed-deadline check
* wdclient: reinterpret only deadline errors as caller-gave-up
isCanceledErr also matches gRPC Canceled, which a filer can return for
its own reasons; treating it as an elapsed caller deadline hid the
failure from the filer's health record. The fallback now matches only
context and gRPC DeadlineExceeded, so filer-side cancellations keep
counting as filer failures.
vector: non-filterable metadata round-trips and filters reject it
- GetIndex omitted nonFilterableMetadataKeys — the catalog stores the
list as a JSON string but the reader only accepted an array; both
forms now decode.
- QueryVectors now rejects filters referencing non-filterable keys
instead of silently scanning them.
* vector: weed-vector registers with the weed master
weed-vector-server --masters holds a KeepConnected stream to the weed
master, the same mechanism filer and s3 use via wdclient — the server
shows up in ListClusterNodes(client_type=vector). Resends registration
and rediscovers on failure.
* vector: s3 discovers weed-vector through the master
With -vectors.grpc unset the vectors gateway resolves one registered
weed-vector via ListClusterNodes on the masters it already derives
from filer config — discovery replaces the static dial for the
single-worker OSS path. The conformance harness registers the engine
to mini's master and drops -vectors.grpc.
* vector: default-fill registration fields
..Default::default() keeps the source compiling against proto variants
that carry extra fields, so the same file works in either tree.
* vector: serve the Vectors API on the main s3 port
The vectors router sits on the main -port listener behind a dispatch
that checks the SigV4 credential scope for /s3vectors/ — no separate
port needed. -port.vectors stays for explicit dedicated-port
deployments; the conformance harness now hits vectors through :8334.
* vector: drop -port.vectors entirely
It was new — no compat to preserve. The main -port listener serves the
API whenever a weed-vector backend is reachable.
* vector: drop -vectors.grpc; discovery is the only locator
weed-vector always registers to master, the gateway always resolves it
through ListClusterNodes — a static address just forks membership.
* vector: binary is weed-vector
Matches the enterprise binary; package and lib names are unchanged.
* vector: review fixes — cluster type, grpc port, lazy resolve, leader hint, advertise
- cluster: register 'vector' as a node type so ListClusterNodes returns
weed-vector members
- s3: dial masters via ToGrpcAddress (HTTP port + 10000)
- s3: backend resolves lazily through vectorDiscovery — workers joining
after gateway startup work on first query, and address changes re-dial
- master_client: convert master HTTP addr to grpc +10000, honor http(s)
scheme, follow the leader hint in KeepConnectedResponse, and
--advertise covers wildcard/unreachable listen addresses
* vector: no --advertise flag; derive the outbound address
A wildcard listen resolves to the source IP the master sees via a
route lookup, so registration always advertises something reachable
without another flag.
* vector: review fixes — plaintext dispatch, conn leaks, discovery backoff
The two plaintext http servers now serve the vectors dispatch too —
signed s3vectors calls on the main port reach the right router.
discoverVectorWorker closes each master conn and bounds lookups to 5s;
a dead or absent worker no longer re-hits master on every RPC — the
resolver backs off for 5s between rediscovery attempts.
* vector: 1 GiB grpc cap, dead-worker invalidation, CLI rename
Valid PutVectors batches exceed the tonic 4 MiB default — match the
gateway's 1 GiB cap. A replaced worker no longer sticks: any RPC error
resets the cached address and the next call rediscovers through the
master. The core CLI bin renames to weed-vector-cli so the two crates
do not collide on the same output path.
* vector: retry-only-transport reset, periodic re-resolve, cli rename
- reset() takes the failed client and runs only on transport errors —
an InvalidArgument on a healthy conn no longer interrupts unrelated
requests, and a concurrent re-resolve wins the race.
- cli() re-resolves once the interval passes even with a live client,
so a worker that moves addresses is picked up.
- verify.rs spike docs use weed-vector-cli after the bin rename.
* vector: only Unavailable counts as a transport failure
A request deadline or an application error can fire on a healthy
connection — resetting the shared conn on those churned reconnects.
vector: keep typed error chains in lease pin/release
Error::msg discarded the source chain — denied() could not downcast to
object_store::PermissionDenied, so try_pin surfaced the error instead
of degrading to an unleased query and read-only credentials broke
GetVectors/QueryVectors. Plain ? preserves the chain.
* wdclient: treat an elapsed caller deadline as caller-gave-up on lookup errors
gRPC can surface its own DeadlineExceeded a tick before ctx.Err() becomes
visible, so a lookup that died on the caller's deadline was sporadically
recorded as a filer failure (seen in the linux/386 job).
* wdclient: reinterpret only deadline errors against an elapsed caller deadline
vector: named_segment_fragments carries the logical index name
Sharded builds create one named index per shard and a
with_index_segments scan must not mix names — a fanout planner needs
the name to group tasks, so expose it alongside uuid and coverage.
vector: keep the full identity in the request context
Auth discarded the resolved identity object — the s3tables manager
request then re-resolved by name and denied authenticated users their
actions, policies, account, and claims. Store the object via
SetIdentityInContext so downstream authorization sees it.
* build(deps): bump com.google.guava:guava in /test/java/spark
Bumps [com.google.guava:guava](https://github.com/google/guava) from 32.1.3-jre to 33.7.1-jre.
- [Release notes](https://github.com/google/guava/releases)
- [Commits](https://github.com/google/guava/commits)
---
updated-dependencies:
- dependency-name: com.google.guava:guava
dependency-version: 33.7.1-jre
dependency-type: direct:production
...
Signed-off-by: dependabot[bot] <support@github.com>
* test/java/spark: bump guava to 33.7.2-jre
33.7.1 is covered by a published advisory for uncontrolled memory
allocation during CompactHashMap/CompactHashSet deserialization;
33.7.2 is the patched release.
* test/random_access: bump guava to 33.7.2-jre
Same deserialization advisory; the pin predates the fix.
---------
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: user.email <chrislusf@users.noreply.github.com>
* helm: render the filer's mysql credentials once when the user sets them
With the MySQL filer store, the filer reads WEED_MYSQL_USERNAME and
WEED_MYSQL_PASSWORD from the chart's db secret. Setting either one in
extraEnvironmentVars or secretExtraEnvironmentVars, as helm_ci.yml does for
the password, added a second entry with the same name: Helm 4 rejects the
install (env: duplicate entries for key), and Helm 3 deploys with a warning
that one definition may be dropped on apply.
The db secret's entry is now left out for a key the user sets, so each key
is rendered once and the user's value wins, the rule #8488 applied to a key
set in both global and filer extraEnvironmentVars. Without an override the
filer StatefulSet is unchanged.
* helm: render the selected filer mysql credential ahead of the merged vars
A credential the user sets now renders in the early position the db
secret's entries used to occupy, and is excluded from the merged and
secret loops. Kubernetes resolves a $(VAR) reference only against names
declared earlier, so a later entry such as WEED_MYSQL_DSN can again
reference the active credential. Secret-backed values win over plain
ones; the db secret's entry remains the fallback.
* helm: let a secret-backed filer env key replace the plain one
secretExtraEnvironmentVars now wins over extraEnvironmentVars for the
same name, rendering once at the merged key's sorted position instead
of twice. The values.yaml WEED_MYSQL_* defaults no longer duplicate
when overridden through the secret map - e.g. enabling the store via a
secret WEED_MYSQL_ENABLED also rendered the "false" default before,
which Helm 4's server-side apply rejects as a duplicate.
* helm: keep a credential that references another env var in sorted position
A user-supplied credential whose value itself holds a $(...) reference
stays in the merged vars' sorted order so its own references still
resolve; only reference-free values, valueFrom entries, and the
db-secret defaults move to the early slot.
* helm: render vars a filer mysql credential references ahead of it
Hoisting the selected WEED_MYSQL_USERNAME/PASSWORD entries above the
merged extra vars let WEED_MYSQL_DSN-style $(...) references resolve,
but it broke the opposite direction: a user-set credential such as
WEED_MYSQL_USERNAME: "$(DB_USER)" was emitted before DB_USER, so
Kubernetes left the reference literal. Pull every merged or secret var
a credential value transitively references out of the sorted set and
emit them first, dependencies before dependents, so both directions
resolve.
* helm: emit filer mysql credentials after the chart's own env vars
The dependency-ordered credential block rendered before
SEAWEEDFS_FULLNAME and the s3 credentials, so a user-supplied
WEED_MYSQL_USERNAME could not reference them via $(...). Move the
block below those fixed entries so references in either direction
resolve.
* helm: bound filer mysql credential ordering passes by var count
Chains deeper than the fixed eight passes forced a credential out ahead
of the vars it references. Cap discovery at the number of merged/secret
vars and emit passes at the ordered list length, so any resolvable chain
finishes before the last-pass fallback emits cyclic leftovers.
---------
Co-authored-by: chrislusf <chrislusf@users.noreply.github.com>
* helm: keep the filer db credentials across uninstall and install
The db secret is kept on uninstall (`helm.sh/resource-policy: keep`), but it
is a pre-install hook, and Helm deletes an existing hook resource before
creating it again. The next install therefore wrote a new random password
and reset the user to YourSWUser, so the filer could no longer log in to a
database set up with the old ones, including credentials edited into the
kept Secret.
The template now reads the existing Secret with lookup and reuses its user
and password; a first install still generates them as before. A kind step
in helm_ci.yml installs, edits the Secret, uninstalls, installs again and
checks both values.
* helm: README, the db secret is kept and reused across a reinstall
* helm: make the db secret hook-delete-policy explicit
before-hook-creation is Helm's default for hook resources, but spell it
out so the reuse contract is obvious: lookup reads the kept Secret at
render time, the old copy is deleted before the hook recreates it, and
the stored credentials carry over.
---------
Co-authored-by: Chris Lu <chris.lu@gmail.com>
Only a string rendered as `value`; anything else went to `valueFrom`. An
unquoted number or boolean in a values file or `--set`, such as
`WEED_MYSQL_PORT: 3306` or `WEED_MYSQL_ENABLED=true`, rendered as
`valueFrom: 3306` and the API server rejected the workload.
A map still renders as `valueFrom`, a null still renders an empty variable,
and strings are unchanged. Numbers go through toJson because values files
decode them as float64, and quote would print 1000000 as 1e+06.
* filer: make the HTTP connection idle timeout configurable
The filer applied a hardcoded 10s activity timeout to its HTTP
listener. A buffering proxy in front of the filer can stall upstream
reads longer than that when the end client drains slowly, so large
downloads through it were truncated (#8062).
Expose the value as -idleTimeout (default 10, unchanged), matching the
existing -idleTimeout flags on the volume and s3 servers, and wire
filer.idleTimeout through 'weed server' and 'weed mini' too. A nil
option keeps the 10s default so embedders constructing FilerOptions
directly keep working.
* filer: apply the idle timeout to the readonly port too
The -port.readonly listener ran without any connection timeout while
the main API port timed out. Give it the same -idleTimeout so both
ports behave identically.
* webdav: make the HTTP connection idle timeout configurable
The WebDAV server used the same hardcoded 10s activity timeout.
Expose it as -idleTimeout (default 10, unchanged), including
-webdav.idleTimeout when embedded via 'weed filer -webdav',
'weed server -webdav', and 'weed mini'.
* iam: make the HTTP connection idle timeout configurable
The IAM API server used the same hardcoded 10s activity timeout.
Expose it as -idleTimeout (default 10, unchanged), including
-iam.idleTimeout when embedded via 'weed filer -iam' and
'weed server -iam'.
* sftp: make the connection idle timeout configurable
The SFTP server used the same hardcoded 10s activity timeout.
Expose it as -idleTimeout (default 10, unchanged), including
-sftp.idleTimeout when embedded via 'weed filer -sftp' and
'weed server -sftp'.
* command: clamp idle timeout seconds before converting to duration
A seconds value past time.Duration's range would overflow on
multiplication and silently disable the timeout. Bound it in one shared
helper, which also deduplicates the nil-default handling at the four
call sites.
* command: return zero for non-positive idle timeout seconds
A large negative seconds value could wrap on multiplication and come
out positive. Guard before converting so non-positive input always
disables the timeout.
* command: apply idle timeout to local unix socket listeners
The unix-socket listeners in filer, s3, and sftp served accepted
connections with no activity deadline while the TCP listeners already
wrapped conns in util.Listener. Wrap the socket listeners the same way
so a positive idleTimeout also bounds idle socket connections.
* command: clamp s3 idle timeout seconds before multiplying
s3 still multiplied the flag inline, so an out-of-range value could
overflow time.Duration. Route it through idleTimeoutSeconds like the
other commands; non-positive keeps meaning disabled.
* volume: clamp idle connection timeout seconds before converting
* command: cover idleTimeoutSeconds conversion semantics
* command: gate idleTimeoutSeconds clamp check on 64-bit int
* s3: answer NoSuchBucket for encryption and policy writes to a missing bucket
PutBucketEncryption, DeleteBucketEncryption, and PutBucketPolicy surfaced
the filer's bucket-miss as InternalError; GetBucketEncryption masked it as
ServerSideEncryptionConfigurationNotFoundError. Map the miss to
NoSuchBucket instead, matching AWS. Internal readers still treat a missing
bucket as "no encryption configured" so uploads keep their behavior.
* s3: wrap ErrBucketNotFound only for verified misses in bucket policy ops
The filer lookup helpers in get/set/deleteBucketPolicy marked every
LookupDirectoryEntry failure as bucket-not-found, so a transient filer
outage answered NoSuchBucket for a live bucket. Route them through
filer_pb.LookupEntry, which normalizes only true misses to ErrNotFound.
* s3: answer NoSuchBucket when the policy delete's own write loses the bucket
* s3: classify a filer NotFound on the policy UpdateEntry as ErrBucketNotFound
* s3: require the bucket entry to exist on config patch transactions
* s3: answer NoSuchBucket when tagging and CORS writes lose the bucket mid-request
* s3: keep the existence check on no-op bucket config patches
An unchanged config write produced an empty patch that early-returned
before the transaction, so a bucket deleted between the read and the
write still reported success. Ship the condition-only transaction so
IF_EXISTS runs on its own.
* s3: guard the iam step and cover failure-vs-miss classification
Nil-guard s3a.iam in the bucket policy handlers so the handlers do not
panic when no IAM server is wired (unit tests, embedded servers), treat
a "precondition failed" response string the same as the error code when
mapping a failed patch to ErrNotFound, and add handler-level tests that
pin the miss/failure distinction: missing buckets and vanished mid-write
buckets answer 404 NoSuchBucket while transient filer failures stay 500
InternalError.
* s3: keep multipart part encryption on the initiate-time decision
putToFiler applied the *current* bucket default encryption when writing a
staged part, so a PutBucketEncryption landing between two UploadPart calls
produced an object whose parts disagree: some chunks ciphertext, some
plaintext. Completion then marks the object by its majority and GetObject
fails decrypting the rest. Parts staged under .uploads now skip the
bucket-default lookup; the decision snapshotted into the upload entry at
CreateMultipartUpload is replayed through request headers instead, which
matches AWS (default encryption applies at initiation).
* s3: mark multipart part writes via request context, not the staged path
Inferring "is a part" from the .uploads file path let an ordinary PutObject
named .uploads/x.part skip bucket default encryption entirely. The part
decision now travels on the request context from the two callers that
stage parts (UploadPart, UploadPartCopy); a client-chosen key cannot reach
it.
S3DisableContentMD5Validation skips the request header that DeleteObjects
requires, so AWS rejects every directory removal and the filer remote sync
offset stalls. Issue the delete through DeleteObjectsRequest and re-enable
the flag on that request's config copy so the SDK's own checksum handler
adds the header without changing any other operation.
Generated with [Devin](https://devin.ai)
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* volume: replicate FetchAndWriteNeedle writes with the configured TLS scheme
The guarded upload path hardcoded an http:// URL and used a bare
*http.Client, so a peer fronted by https rejected the replica write and
read-through of a remote-evicted object failed whenever the volume had
replicas. Normalize the upload URL through the shared global client and
let the guarded transport borrow its https.client TLS configuration, the
same settings the regular replication path resolves.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* volume: keep the global HTTPS downgrade guard on replica uploads
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* volume: cover TLS replicate uploads and the redirect downgrade guard
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* volume: pin the downgrade guard against a dialable plaintext target
The previous redirect pointed at 169.254.169.254, which the guarded dialer
already refuses, so the test passed even with CheckRedirect removed.
Redirect to a live plaintext server on a dialable peer address and assert
it never receives the resent body.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* volume-rs: keep the HTTPS downgrade redirect guard on outbound writes
Mirrors the Go peer upload client: the shared outgoing HTTP client now stops redirects that would downgrade an https request to plaintext, so replicate uploads to TLS peers cannot leak data or auth headers via a 307 to http.
* volume-rs: fail rejected redirects instead of returning the 30x response
Go's rejectHttpsDowngrade uses ErrUseLastResponse, which surfaces to the
caller as a non-2xx status it rejects; attempt.stop() returned the redirect
response as Ok, so a refused downgrade could masquerade as a completed write
to any caller that does not check the status.
* volume-rs: fail replica writes on a non-2xx status
fetch_and_write_needle discarded the replica response without checking its
status, so a peer rejection — or a redirect the client refused to follow —
was reported back as a successful copy while no replica was written. Go's
doUploadData fails on any non-success status.
---------
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* vector: allow restricting ANN search to index segments
Query gains an optional index_segments list; when set and the index
path is used, the scanner restricts the vector index search to those
segments via with_index_segments. This is the seam the Enterprise
planner uses to fan one query out across per-segment workers; OSS
keeps passing None.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* vector: enumerate and partition index segments for fanout
segments::index_segments lists the physical segment UUIDs covering the
vector field at the pinned version; partition() assigns them
round-robin to workers. The parity test proves two workers scanning
disjoint shares merge back to the full-dataset top-K — the P3 exit
gate, in-process.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* vector: let a query carry fragment ids for the unindexed sweep
Lance only applies the index-segment restriction to indexed rows —
unindexed fragments outside the selected segments are skipped. For a
partitioned query to stay correct, the planner needs to hand the
unindexed sweep to a worker too, so Query gains a fragments list that
the scanner resolves to manifest fragments via with_fragments. A task
carrying both searches its segments via the index and its fragments
with flat KNN in the same scan.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* vector: empty scope returns no hits; cover the sweep in tests
Some([]) fragments or index_segments fell through to an unrestricted
scan — a worker assigned nothing would have searched everything.
empty_scope short-circuits those queries, and the segment restriction
only calls with_index_segments on a non-empty list.
segments::segment_fragments also returns each segment's live fragment
bitmap, and the parity test now leaves a fragment unindexed so the
worker carrying it exercises the flat sweep — the contract the
Enterprise planner relies on.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* vector: scope the scan to listed ∪ segment-covered fragments
Lance intersects with_fragments with the selected index segments, so a
worker given its segment share plus the unindexed sweep lost the share
entirely — only the listed fragments were scanned. Expand the fragment
list with the live coverage of the selected segments. The list is now
also applied when it resolves empty: stale ids or an empty assignment
must scan nothing, not fall through to an unrestricted scan. The merge
test now leaves a fragment unindexed so the combined path is exercised.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* vector: partition assigns nothing when there are no workers
n.max(1) returned every segment in one phantom partition for a
nonexistent worker, silently dropping coverage.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* webdav: refuse MKCOL and PUT under a missing folder with 409
RFC 4918 9.3 and 9.7.1 say a MKCOL or PUT whose parent collection does
not exist must fail with 409 Conflict. The filer creates missing parents,
so both succeeded and made the intermediate folders. Check the parent in
Mkdir and in OpenFile with O_CREATE and return os.ErrNotExist, which
x/net/webdav turns into 409.
* webdav: check the PUT parent before an overwrite deletes the target
OpenFile deleted an existing destination before the new parent lookup,
so a PUT whose parent check failed left the old file gone.
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* webdav: report a missing rename source as not-exist
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* webdav: refuse MOVE to a missing destination folder
AtomicRenameEntry grows any missing parents itself, so a MOVE to
/missing/file created /missing; check the destination parent first.
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* webdav: answer 409 for COPY and MOVE under a missing collection
The webdav handler reports a failed directory Mkdir or Rename as 403,
so the destination parent is checked at the request layer instead.
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* webdav: defer foreign-host MOVE and COPY destinations to the 502 handler
* webdav: only check the destination parent for absolute paths
* webdav: cover foreign-host and relative MOVE destinations in the guard test
---------
Co-authored-by: user.email <chrislusf@users.noreply.github.com>
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
checkAccessByOwnership compared the caller's account with the bucket
owner even with no identities configured, so Get/Put/Delete
BucketOwnershipControls answered AccessDenied to every request. With
auth off there is no owner to compare against, as the other auth-off
checks already assume.
The pinned dev snapshot (v1.85.0-dev.0.20260915183914) predates
grpc/grpc-go#9563, which fixes the control-buffer throttle deadlock
(grpc/grpc-go#9330) that wedges weed mini under concurrent S3 writes
over unix sockets.
Verified with the issue's standalone repro (one ClientConn over a unix
socket, concurrent unary RPCs): the old snapshot deadlocks at c=512
after 743 RPCs; the new snapshot completes 3.06M RPCs/20s at c=512 and
1.97M/15s at c=2048 with no stall -- above the v1.82.0 pre-regression
baseline of ~112k ops/s.
* topology: keep the oversized verdict after the location list is gone
The per-replica oversized marks live in the volume's location list, which
is deleted when its last location leaves and recreated flagless on the
next registration. Between that recreation and the next heartbeat that
re-measures the volume, ensureCorrectWritables saw no mark and
re-admitted an oversized volume to the writable list; two other paths
(quarantine release, mark-writable) never consulted the marks at all.
VolumeLayout now remembers the oversized verdict per vid, written only
by heartbeat sizes and forgotten only when a report measures the volume
under the limit. The writable admission gate itself refuses a marked
volume, and the vacuum commit path drops a verdict its live size check
has just disproved.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* topology: mark oversized at registration when the report carries a size
RegisterVolumeLayout ran EnsureCorrectWritables while the recreated
location list still held no per-replica marks. When the registering
report knows the volume's size, refresh the oversized state before the
writable check instead of waiting for the next heartbeat pass.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* topology: test that oversized volumes stay out of writables across re-registration
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* topology: remember the oversized verdict per replica
A shared per-volume verdict was cleared by any single replica's
under-limit report, so a still-oversized replica whose location entry
had died could re-register from a size-less report and put the volume
back on the writable list. Keep one verdict per reporting replica: an
under-limit report clears only its own, and a replica that never
returns cannot pin its volume out of writables forever because the
verdict ages out. The vacuum commit now feeds the node's fresh answer
through the same per-replica bookkeeping instead of erasing the whole
verdict.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* topology: count a volume at the size limit as full on vacuum commit
The layout's oversized test is >= the limit, but the vacuum commit and
status checks used >, so a replica sitting exactly at the limit passed
as not-full and the commit path could re-admit it.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* topology: cover per-replica oversized verdicts in tests
One replica shrinking must not clear a still-oversized replica's
remembered verdict; cover that plus a size-less re-registration before
the second replica is re-measured. The vacuum-commit case now shrinks
the stored record first, since the commit re-derives the mark from the
node's own answer.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* topology: judge the vacuum commit by the size it just measured
SetVolumeAvailable rebuilt the oversized verdict from the cached
VolumeInfo, which still carries the pre-compaction size until the next
heartbeat. A commit that shrank a volume under the limit re-marked the
replica oversized and never restored writes. The commit path already
asks every replica for its fresh size; pass it through per replica and
record it on the mark, the verdict, and the size tracking seed.
* topology: hold an expired oversized verdict while its replica is back
The TTL forgives a replica that never returns, but a remount announces
the volume without a size and registration skips the oversized check.
Deleting the verdict on expiry then re-admitted a still-oversized
volume until the next size-bearing heartbeat. Keep counting an expired
verdict while its replica sits in the live location list, and keep
sweeping verdicts whose replicas stayed gone.
---------
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: chrislusf <chrislusf@gmail.com>
calculateClusterCapacity multiplies advertised volume slots by the
volume size limit. Slot counts are an admission bound rather than a
capacity measurement — an explicit -max can be set higher than the
volumes a disk actually fills — so a small disk can report more total
and available capacity than it physically has. Bound both numbers by
the per-disk filesystem counters the volume server already reports in
its heartbeat, falling back to the slot estimate only when they are
absent.
Generated with [Devin](https://devin.ai)
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* mount: drain OFD locks when a file description is released
The kernel scopes OFD fcntl locks to the open file description but
reports no lock_owner for them at RELEASE — it only sends one for flock
holders — so an OFD write lock taken on a file handle outlived the
holder and blocked every later open of that file (e.g. etcd failing to
reopen its WAL after a restart). The mount now records the owner of each
granted non-flock lock under its file handle and releases those owners
on RELEASE, which also covers any POSIX locks left over when FLUSH could
not run.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* mount: give every open its own fh value over the shared handle
AcquireFileHandle reuses one FileHandle per inode, so every open reported
the same fh to the kernel. A RELEASE for one description could then not
be told apart from a sibling still holding the inode open — draining
lock owners recorded under the shared fh dropped locks belonging to
descriptions that stay open.
Each open now registers its own fh value in fh2inode, all resolving to
the shared FileHandle, and RELEASE removes exactly the id that is
closing. This also stops a duplicated RELEASE on one id from
double-decrementing the open count.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* mount: forget flushed lock owners so a later release cannot re-drop locks
POSIX lock owners are process-scoped: Flush releases every lock the
process holds on the inode, but the owner stayed recorded under each
open's fh for the Release-time drain. If the process took a fresh lock
through another open before an old description finished closing, the
stale record made that RELEASE drop the new lock too.
Flush now drops the released owner from every recorded open of the
inode, so the Release drain only ever fires for owners whose locks were
never flushed.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* mount: keep close-time lock owner records when the release fails
A routed POSIX owner release that errors keeps the flush hint for a
retry, but Flush also dropped the owner from every open's release
records unconditionally — so the final Release skipped the retry and
the owner filer's lock was stranded until lease reaping. Only forget
the owner once its locks were actually released.
---------
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* cluster: bound each DistributedLock attempt with a timeout
An unreachable lock owner blocked the attempt on the connection timeout.
Bound it so the write fails fast; a timed-out attempt keeps no renew token.
* cluster: keep renewToken across failed lock attempts
A request may have renewed the lease server-side before its response was
lost, so the last token can still be valid on the next attempt. Clearing
it on any transport failure strands a live lease until its TTL expires.
A definitive rejection still clears the token through the response's
empty RenewToken, and falling back to the seed filer no longer abandons
the lease either.
* cluster: bound each lock attempt by the remaining route window
The 15s unavailability window started only after the first failure, so
three 10s attempts plus sleeps outlasted it: the third failure selected
the seed fallback at the same moment the loop gave up, and a reachable
seed route was never tried.
Start the clock before the first attempt and cap each attempt at what
the window has left. When the window is spent on a non-seed target,
fall back to the seed filer — it can forward to the real owner — and
give that route its own window, so the fallback is always exercised
before acquisition fails.
* cluster: bound DistributedUnlock by the lock RPC timeout
The unlock used context.Background(), so Stop against an unreachable
lock server could wait on the TCP connect timeout far past the renewal
goroutine's bounded shutdown wait. A timed-out unlock is harmless: the
lease expires via its TTL regardless.
* dlm: replay the current token on a lost renewal's retry
A renewal rotates the token server-side before the response is sent. When
that response is lost the client only knows the superseded token: retries
mismatched, the token was cleared, and the holder conflicted with its own
lease until it expired. The lock now remembers the superseded token, so a
retry from the same owner replays the current token, and a superseded
token can still release its own lock.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* cluster: give the seed lock fallback one fresh window per acquisition
The seed can redirect back to a dead primary via LockHostMovedTo, so
refilling the retry window on every return to the seed made the
primary->seed->primary cycle unbounded. The first seed fallback keeps
its fresh window; later returns to the seed keep the running deadline.
* cluster: advance the lock seq when a lost-renewal replay extends the lease
A retry of a renewal whose response was lost updates ExpiredAtNs in place
but returned the existing Seq, so the replicated extension was rejected
by InsertBackupLock as a stale mutation. A promoted backup would then
grant the lock when the old expiry passed even though the holder still
believes the lease is renewed. Bump Seq on the replay so the backup
accepts the new expiry.
* cluster: carry the superseded lock token through replication and transfer
A renewal rotates the lock token and keeps the superseded one in
PrevToken so a holder whose renewal response was lost can still recover.
That recovery broke as soon as the lock moved: neither the replicated
backup copy nor a transferred lock carried PrevToken, so a new primary
rejected retries of the old token until the lease expired.
Add prev_renew_token to the Lock and ReplicateLockRequest messages and
thread it through ReplicateLock, TransferLocks, InsertBackupLock and
InsertLock.
* cluster: count clean lock redirects against the retry window
A bare LockHostMovedTo response (no error, no token) returns nil from
attemptToLockWithin, and retryUntilLocked reset the retry window on every
nil result. Two ring-disagreeing filers forwarding to each other therefore
bounced the client in an unbounded loop; the spent-window exit could also
return a nil error with no token, handing callers an unacquired lock.
* cluster: do not mark a lock held or hot-loop on tokenless attempts
A bare LockHostMovedTo redirect returns nil with no renewToken; the
attempt previously flipped isLocked to 1 anyway and retried immediately.
Gate the flag on a granted token and pace the retry like other outcomes.
* cluster: keep renewToken across bare lock redirects
A LockHostMovedTo response carries no token and no error; overwriting
renewToken with it dropped a held token, so the next renewal re-acquired
fresh and could contend against its own still-live lease until TTL.
* cluster: poll for replicated backup state in lock tests
Replication runs in a goroutine, so a fixed sleep followed by an
immediate backup read is racy. Wait for the expected backup state
with require.Eventually instead.
* cluster: mark a lock held only on a grant verdict
A bare LockHostMovedTo redirect with a saved token left isLocked set
without the redirected filer granting anything, and retryUntilLocked
could exit on a stale token with no attempt made. doLock now reports
whether the response was a grant, and both the retry loop and isLocked
follow that verdict.
---------
Co-authored-by: chrislusf <chrislusf@users.noreply.github.com>
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* vector: address remaining s3vectors review findings
Catalog and permission model:
- Vector buckets are declared Lance table buckets end to end: create sets
format=lance, get/list/delete reject or skip ICEBERG buckets, and the
format lookup runs on an internal trusted catalog so callers need no
GetTableBucket grant.
- The implicit "vector" index namespace is created through the same
internal manager, so a CreateTable grant alone authorizes CreateIndex;
typed S3TablesError codes replace string matching on err.Error().
- CreateIndex pre-checks existence after the authorize, tags the record
with a create nonce, and rolls back on engine failure under
context.WithoutCancel with a versionToken CAS, so a retry never deletes
a table it did not create.
- DeleteIndex drops the catalog record directly; DropIndex is
unimplemented on the shared VectorService.
- DeleteVectorBucket removes the empty internal namespace on
BucketNotEmpty and retries once.
- PutVectors/DeleteVectors require the update grant, and engine gRPC
errors map to typed API errors instead of generic 500s.
API correctness:
- CreateVectorBucket/CreateIndex return their resource ARNs.
- ARNs reject empty bucket segments; vectorBucketArn must be an ARN.
- ListIndexes returns summaries built only from catalog fields.
- ListVectors honors maxResults/nextToken inside a segment via
"seg:<pageStart>:<nextIndex>" resume tokens and validates
segmentCount/segmentIndex.
- QueryVectors requires positive topK; PutVectors rejects empty keys.
- CreateVectorBucket rejects a supplied encryptionConfiguration rather
than silently dropping it.
- Metadata filters emit Lance's real JSONB UDFs: json_get_string/
json_get_float/json_get_bool picked by operand type, json_exists with a
bracketed JSONPath, and per-member accessors for $in/$nin.
Engine:
- weed-vector-server stores the metadata column as Lance JSONB
(lance.json extension) so the json_* UDFs apply; reads accept legacy
Utf8 JSON columns.
Vectors port TLS:
- The TLS config is built before binding listeners, loads -cacert.file
into ClientCAs when client-cert verification is on, fails startup on an
empty/invalid CA file, and closes the gRPC conn on every failure path.
* s3vectors: authorize index create before writing the implicit namespace
A denied caller could still leave a persistent namespace behind because
ensureIndexNamespace wrote through the trusted catalog before the
caller's create grant ran. AuthorizeCreateTable now tolerates the
not-yet-declared implicit namespace (bucket policy alone decides) and
runs before any write; real table creates keep requiring the namespace.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* s3vectors: keep engine error details out of the API response
Unavailable now maps to 503 ServiceUnavailableException and unclassified
codes return a generic message; the real gRPC error stays in the log.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* s3vectors: exact-type metadata filters + NULL-safe $nin
The jsonb extractors convert numeric-looking strings through
json_get_float and 'true'/'false' through json_get_bool, so a stored '3'
could satisfy a filter for the number 3. Numeric and boolean comparisons
now also require the stored value not be a string, and each $in/$nin
group coalesces NULL so NOT cannot invert NULL into NULL.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* s3: validate the vectors TLS key pair before binding
The reloading certificate provider reads the files asynchronously, so a
missing or mismatched pair would only surface as failed handshakes on an
announced HTTPS port. Load it once up front.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* s3tables: make a token-checked table delete atomic
The version token was read in one filer call and enforced in another, so
a delete+recreate slipping between them let the delete erase the
replacement. When a token is supplied the name path now carries the
observed mtime as an if_not_modified_after precondition (evaluated under
the path lock), and a decoupled delete strips the catalog marker under a
compare of the captured attributes before touching the data path.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* s3vectors: compare filter operands against the stored JSON type
jsonb's strict converters make json_get_string return text for numbers
and booleans too, so a get_string IS NULL guard is never true for scalar
metadata — numeric and boolean comparisons silently matched nothing.
Numbers and booleans now compare through TRY_CAST of the serialized
json_extract text, which yields NULL on any other stored type; string
comparisons stay on json_get_string behind a CASE guard so the extractor
only runs on rows storing a JSON string.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* s3tables: allocate implicit namespace metadata before use
A first table create in an undeclared namespace took the
allowMissingNamespace branch with a nil record and panicked on the
owner assignment.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* s3tables: read the delete target's metadata and snapshot in one lookup
Two separate lookups let a delete+recreate in between validate the
caller token against the old record while the conditional delete aimed
at the replacement.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* s3tables: assert the captured record before a conditional table delete
An xattr update keeps Mtime, so the IfNotModifiedAfterNs guard alone
could not see a metadata write that had already rotated the version
token, and a recreate preserving the mtime could slip the mtime check.
The delete now first proves the entry still carries the captured
attributes under the filer's path lock, then keeps the mtime
precondition.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* s3vectors: match only JSON boolean literals in metadata filters
* filer,s3tables: delete versioned tables under one attribute precondition
* s3vectors: compare serialized JSON for string filter operands
json_get_string converts numeric-looking strings and booleans, so it
cannot serve exact-typed string comparisons even behind a guard. String
operands now compare the serialized json_extract text directly, which is
exact-typed and never errors; a Rust test pins the semantics against a
mixed-type dataset.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* s3vectors: compare boolean filter operands as serialized text
Arrow's boolean cast accepts '1' and '0', so a TRY_CAST column let a
stored number satisfy a boolean filter. Booleans now compare the
serialized json_extract text like strings do — a JSON boolean only ever
serializes as 'true' or 'false'.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* s3tables: claim a catalog name's markers in one compare-checked write
A create wrote its markers attribute-by-attribute after checking the
name was free. A delete's restore landing in the check-to-write gap put
the old markers back first, and the create then overwrote only the ones
it sets — leaving the old tags, policy, and maintenance settings on the
new table. CreateTable, RegisterTable, and CreateView now write all
their markers in a single update that refuses to run while any catalog
marker is present, so the second writer loses with a conflict instead
of inheriting them.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* s3tables: test the attribute-preconditioned table delete and name claim
Adds a BeforeDelete hook plus ExpectedExtended evaluation to the
in-memory filer so tests can land a competing write in a handler's
check-to-delete window, and covers two losing races: a token delete
whose record changed since the read, and a create colliding with a
delete's marker restore.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* s3tables: reserve a catalog record under one compare-checked claim
claimCatalogRecord writes a deleting marker onto the record while it
still holds exactly the caller's captured attributes, so a delete or
rename can hold the record through a multi-step operation and a losing
writer backs out before any data or destination entry is written. The
claim key joins the markers a name-claim asserts absent, so the name
cannot be reused while the reservation stands.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* s3tables: claim the record before purging or publishing past it
A token delete of a decoupled table purged the data first and checked
the marker afterwards, so a rename that finished in between left the
moved record pointing at purged data while the delete reported a
conflict. The delete now claims the record under the captured
attributes before any data is removed and releases the claim if the
purge fails, keeping the record intact for a retry.
The rename claims the source before publishing the destination, so a
losing rename backs out while no destination entry exists instead of
leaving one behind pointing at data a delete went on to remove.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* s3tables: test the record claim across delete and rename races
A rename completing in the token delete's check-to-claim window turns
the delete into a conflict with the data intact, and a delete's claim
on the source turns the rename into a conflict before the destination
entry is published.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* vector: version-lease helpers + maintenance safety tests
Pinned queries must survive Lance maintenance. maintenance::pin tags the
current dataset version so cleanup refuses to reclaim it; release drops
the tag. Tests prove the P2 gate: a pinned query returns identical hits
across compact_files, a tagged version survives cleanup_old_versions
(error_if_tagged protects it, the non-strict run skips it), and a
missing data file fails the query instead of returning partial hits.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* vector: lease the query's dataset version in the VectorService
query_vectors pins the resolved version with a query tag for the
duration of the RPC and releases it after, so concurrent maintenance
cleanup cannot reclaim the files out from under an in-flight read.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* vector: release the version lease on every query exit path
The lease tag was only released on query_vectors' success path — get_vectors
never released and query_vectors leaked its tag on early returns. pin now
returns a guard whose drop deletes the tag, so every path frees the version.
* vector: unique lease tag names + reap abandoned leases
Nanos-since-epoch tag names can collide across concurrent readers and
processes, so each lease now carries a unique suffix. pin() also drops
query tags older than an hour, since a killed or cancelled request
leaves its lease behind and would otherwise block cleanup forever.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* vector: keep live lease tags + throttle abandoned-lease scans
An expired tag is only reaped once no live lease in this process owns
it, so a query running past LEASE_TTL keeps its pin. Tag scans run at
most once per 60s per dataset instead of on every request.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* vector: answer read-only queries without a version lease
A namespace may hand the server credentials that can read a table but
not update it, so the tag write in pin() fails and the whole query
errors out. try_pin() degrades a denied tag write to an unleased query
instead of failing the request.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* vector: fail a lease whose pinned version is already gone
Cleanup lists tags once at the start, so a tag created mid-cleanup does
not protect the version it names. After the tag lands, pin now verifies
the version still exists and drops the lease when it does not, failing
the read at pin time instead of mid-query.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
test: assert the resumed segment token
The second listSegmentVectors result was never checked (staticcheck
SA4006); assert it resumes scanning at row 19 after matches 10..18.
* filer.remote.gateway: skip uploading a dropped local copy on update
The bucketed update path uploaded the chunkless entry whenever the update
was not metadata-only. remote.uncache, eviction, and remote.meta.sync drop
a synced entry's chunks and clear its sync stamp, which reads as dirty
when the remote mtime shares the entry's second; the upload then replaced
the object that is the entry's only content with zero bytes. Skip the
upload the same way the mounted-directory sync does.
* filer.remote.gateway: copy a remote-only entry's object on rename
A rename deleted the source object and uploaded the chunkless entry,
replacing the entry's only content with zero bytes — the same loss
remote.uncache caused on plain updates. Complete the rename on the
remote the way the mounted-directory sync does: copy the source object
over (or take the destination when it already describes the entry),
restamp so reads stop resolving through the pending-source marker, and
remove the source only while it is still the object the stamp describes.
A remote-only entry renamed out of the gateway's coverage keeps its
object, and the plain rename's old-key delete only removes the object
the event's stamp still describes.
* filer.remote.gateway: scope the sync-stamp skip to same-path updates
shouldSendToRemote reads the entry's sync stamp, which still covers a
rename's unchanged mtime, so a synced or pending-stamped rename event was
skipped before its copy or upload could run. Only updates in place may
skip on the stamp.
* filer.remote.gateway: keep the source a pending rename moved on from
A rename event whose pending destination has since been renamed again
still names the source a later event reads through its own marker;
deleting the old key here loses the only copy. Mirror the remote.sync
check: leave the source when the event's entry carries the pending-source
marker and no entry holds the destination any more.
* filer.remote.gateway: release a rename's pending source once complete
A rename whose old entry read through a pending-source marker leaves that
earlier source behind: the destination is released by its own key, never
the marker's. Mirror releaseRenamedSource — once the renamed entry no
longer reads through a marker, delete the pending source guarded by the
old entry's stamp, keeping it while a remote-only entry holds its path.
* filer.remote.gateway: finalize a complete rename before failing its source
A rename source outside the gateway's coverage errored before the
destination was checked, and the recorded failure held the event offset.
Stat the destination first: when it already holds the content the entry
describes, refresh the stamp and leave the unmanaged source untouched;
fail only when nothing holds the content.
* filer.remote.sync: document the source-of-truth protocol both processors share
---------
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test/vectors: give the vector server static storage credentials
The e2e harness ran weed-vector-server without credentials, so lance's
AWS object store fell back to IMDS and CreateIndex died on the runner's
411 from the token endpoint. Point it at the test credentials the s3
gateway now reads from s3.json.
* vector: take the list item field from the schema when batching vectors
Lance stores the fixed-size list item as nullable even when the write
schema declared it non-null, so PutVectors built batches that no longer
matched: expected FixedSizeList(4 x Float32) but found FixedSizeList(4 x
non-null Float32).
* vector: store index metadata as lance jsonb so filters resolve
The metadata field was plain Utf8, but the filter functions lance's
planner registers (json_get_string, json_get_float, json_exists) take a
LargeBinary jsonb column, so QueryVectors with a metadata filter failed
to plan. Declare the field with the arrow.json extension — lance encodes
it to jsonb on write and back on read — and encode the batch's strings
when the stored column is LargeBinary. Also rename the filter's
json_get_str call to json_get_string, the name lance registers.
---------
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* vector: boto3 s3vectors conformance harness
test/vectors spins a mini cluster, an s3 gateway with -port.vectors and
-vectors.grpc, and a weed-vector-server, then exercises the AWS client
end to end: bucket and index lifecycle, put/get/list/delete vectors,
and a filtered query asserting distances and metadata.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* ci: run the s3vectors conformance subset
An e2e job in rust-vector-tests: builds weed and weed-vector-server,
then runs test/vectors against a mini cluster plus the vectors gateway.
The workflow now also watches vector.proto, its stubs, and test/vectors.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* ci: fix e2e protoc and codespell findings
The e2e job needed the same vendored-protoc step as the unit-test job —
lance's build scripts locate protoc themselves. Codespell flags the AWS
`$nin` operator literal and the `lits` variable: exempt `nin` in
.codespellrc like the existing word exemptions, rename the variable,
and fold the negation into the IN/NOT IN symbol directly.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* s3: classify catalog errors by code, not message text
* s3: drop the implicit vector namespace on vector-bucket delete
* test: point the vector engine at the gateway lance port
* test: bind the vectors gateway listeners to loopback
* test: make the filter and distance assertions load-bearing
* test: rebuild the harness binaries on every run
* ci: fix the vectors end-to-end job
* ci: run the vector tests on gateway and command changes
---------
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* s3: nil-check metadataConfiguration before reading its keys
* s3: honor maxResults/nextToken in segmented ListVectors
* s3: exclude non-Lance buckets from ListVectorBuckets
* s3: test segmented ListVectors paging token math
* ci: silence codespell on the "$nin" operator and rename lits
* vector: S3 Vectors API gateway over VectorService
weed/s3api/s3vectors serves the AWS S3 Vectors JSON API: bucket CRUD
maps to S3 Tables table buckets, index CRUD to Lance table records plus
the VectorService CreateIndex/DropIndex RPCs, and the data plane
(Put/Get/List/Delete/QueryVectors) calls the shared service. The AWS
filter document translates to a Lance SQL predicate over the metadata
JSON column; unsupported operators fail loudly rather than overmatch.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* vector: -port.vectors and -vectors.grpc on weed s3
Mirror the Lance server startup: the S3 Vectors API binds its own port
and dials the weed-vector-server gRPC address; both disabled by default.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* vector: resolve index ARNs, authorize index ops, fix catalog fields
- vector buckets declare FormatLance so a new bucket accepts index
tables instead of defaulting to ICEBERG.
- ListIndexes decodes the `tables` response field; `tableSummaries`
hid every index.
- resolveIndexARNs honors indexArn alone; handlers no longer treat
indexName as the only carrier.
- data-plane RPCs fetch the index record first, so callers authorize
against the index resource and a missing index 404s before touching
the engine; QueryVectors reports the index's configured metric.
- IndexArn added to the index-level request types.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* vector: serve the vectors port over TLS when configured
When -key.file/-cert.file are set, the vectors listener mirrors the
main API and serves TLS rather than a second plaintext endpoint.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* vector: harden the s3vectors gateway handlers
- $ne no longer shadows the operator: exclusion filters emit <>
instead of equality.
- a null element in a PutVectors list fails the request instead of
panicking.
- requests are capped at 32 MiB; distances return via *float32 so an
exact match still reports 0 when returnDistance is set.
- CreateIndex declares the `vector` namespace first and validates
dimension (1..=4096) before the catalog write; a failed
VectorService.CreateIndex rolls the table record back.
- non-filterable metadata keys persist as JSON rather than a
comma-joined string, and index records carry their createdAt.
- ListVectors honors segmentIndex/segmentCount by slicing the full
listing in the gateway.
- catalog errors map to AccessDeniedException/ConflictException instead
of a blanket ValidationException, and malformed ARNs 400 instead of
slicing.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* vector: /{Operation} routes and vectors-port TLS, conn cleanup
Path-dispatched clients (POST /QueryVectors) reach the same handlers as
X-Amz-Target calls. A failed listener bind now closes the vector gRPC
connection instead of leaving it open with the endpoint disabled.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* ci: retry the timing-sensitive s3 lifecycle expiration tests once
test_lifecycle_expiration and test_lifecyclev2_expiration assert exact
object counts after fixed sleeps. The lifecycle worker expires objects
via meta-log replay, and under the suite's write flood a shard's replay
can trail an object's PUT event past the test's window (observed on
master and PR runs, SQL-store job only). Split them out of the main
list and allow one retry; each attempt is independent so real
regressions still fail.
* ci: allow a second retry for s3 lifecycle expiration tests
* s3: list every bucket when auth is off
With no identities configured, every request may create, read and
write any bucket, but ListBuckets treated the caller as an
unauthenticated user and returned an empty list, so `aws s3 ls` showed
nothing. List all buckets in that mode, with prefix, max-buckets and
continuation tokens as before. Listing with auth on is unchanged.
* s3: cover the auth-on anonymous ListBuckets case
The no-auth listing change also moves the nil-identity gate; pin the
security-relevant direction that a request without an identity still
sees no buckets once auth enforcement is on.
* s3: verify ListBuckets continuation token reaches the next page
---------
Co-authored-by: user.email <chrislusf@users.noreply.github.com>
* vector: VectorService protobuf contract
The shared RPC boundary between the Go S3 gateway and the Rust vector
engine from the vector design doc: CreateIndex/DropIndex, Put/Get/
List/Delete/Query vectors. A Table reference carries the Lance table id
and an optional pinned version, matching the snapshot discipline queries
are required to keep.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* vector: generated Go stubs for vector.proto
protoc output via weed/pb/Makefile.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* vector: carry return_data through QueryVectors
S3 Vectors QueryVectors honours returnData, but the contract had no
way to ask for or carry vector data in a query response, so the
gateway would need a second GetVectors round-trip per query. Add
return_data to the request and data to QueryResult now, while the
contract is still new.
* vector: pin metadata numbers to double precision in the contract
Metadata arrives as JSON, where numbers carry double precision; say
so on the field so neither side promises integer fidelity past 2^53.
* vector: declare_table on the lance namespace client
CreateIndex needs a way to ask the catalog for a new table's location;
expose POST /v1/table/{id}/declare.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* vector: cargo fmt coverage.rs
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* vector: weed-vector-server, the VectorService gRPC
A tonic server over weed-vector-core: CreateIndex declares the table
through the namespace and writes an empty {key, vector, metadata}
dataset; PutVectors upserts by key (delete + append until merge_insert
is exercised); Get/List/Delete/Query route through the engine's
resolved table, honoring pinned versions and the prefilter default.
Metadata travels as a Utf8 JSON column, proto Struct on the wire.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* vector: stamp the metric in the schema, tighten put validation
CreateIndex now bounds dimension at the S3 Vectors 4096 maximum and
records the distance metric in the vector field's Arrow metadata, so a
later reader recovers the index's configured metric from the table
itself. batch_of checks every vector's data length individually before
flattening — equal totals could still let a short vector borrow floats
from its neighbor — and writes against the dataset's own schema.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* vector: upsert vectors in a single merge commit
PutVectors ran delete-then-append: a failed append left the old keys
deleted, and two concurrent puts could both append a replacement. Use
lance's merge_insert on the key column — one commit, no delete window,
conflicts fail the request instead of duplicating keys.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* vector: read the index's metric; fetch metadata in one take
QueryVectors hardcoded L2 — it now reads the metric the index was
created with from the vector field's metadata. return_metadata fetched
each hit's row one at a time; collect the row ids into a single
take_rows call and join metadata back on the hit key.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* vector: honor return_data in query results
QueryResult grew a data field but query_vectors never populated it.
Fetch the vector column in the same batched take_rows as metadata and
return each hit's stored embedding.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* vector: reject a malformed query as invalid_argument
A zero top_k silently became one result and a dim-mismatched query
vector surfaced as an internal scan error; reject both before querying.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* vector: reject a malformed list next_token
A token that failed to parse silently restarted the listing from
offset zero; surface it as invalid_argument instead.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* vector: answer an empty put without a commit and cover ragged input
An empty vectors list would still run a merge commit; return the
current version instead. Pin the ragged-vector check with a test.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* vector: keep non_filterable_metadata_keys on the table schema
create_index dropped the declared filter policy. Persist it as JSON on
the metadata field so the index carries which keys a filter may touch;
enforcing the list belongs to the gateway that translates the request
filter into Lance SQL.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* vector: reject duplicate keys in one put
A repeated key for a new row committed both copies, and for an existing
key the merge failure surfaced as internal. Reject duplicates up front
as invalid_argument.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* vector: report open failures as not_found or unavailable
A catalog 404 or a missing dataset/version is not a client mistake, and a
catalog outage is retryable: mapping every open error to invalid_argument
made both indistinguishable from a bad request.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* vector: page list_vectors when max_results is unset
A proto3 client that omits max_results sends 0, which used to limit the
scan to zero rows and return no token.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* vector: drop stray empty service.rs.tmp
* vector: rename datas map to data_map
codespell flags datas as a misspelling of data; use a distinct map name.
---------
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* vector: VectorService protobuf contract
The shared RPC boundary between the Go S3 gateway and the Rust vector
engine from the vector design doc: CreateIndex/DropIndex, Put/Get/
List/Delete/Query vectors. A Table reference carries the Lance table id
and an optional pinned version, matching the snapshot discipline queries
are required to keep.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* vector: generated Go stubs for vector.proto
protoc output via weed/pb/Makefile.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* vector: carry return_data through QueryVectors
S3 Vectors QueryVectors honours returnData, but the contract had no
way to ask for or carry vector data in a query response, so the
gateway would need a second GetVectors round-trip per query. Add
return_data to the request and data to QueryResult now, while the
contract is still new.
* vector: pin metadata numbers to double precision in the contract
Metadata arrives as JSON, where numbers carry double precision; say
so on the field so neither side promises integer fidelity past 2^53.
* vector: clarify the query filter is raw Lance SQL over one metadata column
The doc said 'metadata columns' but the table stores all metadata in a
single JSON column named metadata; the gateway translates S3 Vectors
filter expressions onto it.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* filer: shut down test filers' meta-log buffers, fixing the 32-bit OOM
Each log_buffer.NewLogBuffer preallocates five 8 MiB windows and spawns
loopFlush/loopInterval goroutines that keep them reachable. The
weed/filer tests built ~70 filers and aggregators that almost never
called ShutdownLogBuffer, pinning ~2.8 GB plus live churn, and the
linux/386 job ran out of its ~4 GiB address space inside
newSealedBuffers.
- MetaAggregator.Shutdown now retires MetaLogBuffer too: the "aggr"
buffer's loops previously outlived the aggregator.
- newTestFiler and newTestFilerWithStubStore register t.Cleanup
shutdowns; the one-off filer/aggregator literals defer theirs.
* filer: wait for the aggregated meta-log buffer to stop in Shutdown
ShutdownLogBuffer only signals the stop; loopFlush and its pooled windows
exit asynchronously. Match Filer.Shutdown's local-buffer sequence so the
aggregator returns with the buffer's memory already released.
* filer: cancel test filers' flush context before buffer shutdown
The stub-store helpers build real NewFiler instances, so the queued
shutdown flush retries appendToFile every ~737ms under flushCtx, which
nothing ever cancelled — the buffer and its pool stayed alive past each
test. Cancelling first lets the flush drop the window and WaitForShutdown
confirms loopFlush exited.
* filer: wait for test log buffers to drain after shutdown
The stub-flush buffers cannot hang, but without WaitForShutdown their
loopFlush goroutines release the pooled windows asynchronously — the
memory this PR means to free is still counted as live while the next
test allocates.
* vector: --vector takes hyphenated floats again
* vector: --vector-column/--key-column on the get subcommand
* vector: --vector-column/--key-column on describe and coverage
* vector: surface describe resolve errors when column flags are set
* vector: cover the restored CLI flags with parse tests
* filer: deliver meta-log events under their stream position, not their stamp
A mutation's event timestamp is stamped before the filer store write so
the entry and its event share a commit time, while the log-buffer append
happens after the write. A write delayed behind later-stamped mutations
keeps its early stamp in the payload while AddDataToBuffer bumps
LogEntry.TsNs to keep the stream monotone - so subscribers could
receive events whose payload TsNs arrives out of order.
Consumers treat the delivered TsNs as a stream position (cursors,
watermarks, replay windows). The s3 lifecycle daily-run drains events in
order and pins its per-shard cursor behind the first not-yet-due match;
an event delivered with a stale stamp after the cursor already passed it
is filtered out forever and its pending match silently never fires -
the flaky test_lifecyclev2_expiration signature of exactly one object of
a same-rule pair surviving, with errors=0 on every pass.
Normalize event.TsNs to logEntry.TsNs at delivery (and for the
skeleton-filtered heartbeat) so delivered timestamps are monotone by
construction. The commit-time stamp remains available on the entries
themselves via CommitTsNs.
* pb,filer: stamp metadata events with a mutation timestamp
TsNs on the wire now carries the delivery stream position, so the
mutation's commit stamp needs its own field. Add mutation_ts_ns to
SubscribeMetadataResponse and set it when the event is created.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* filer: carry the mutation stamp through event delivery
Delivery rewrites TsNs to the stream position; the payload stamp survives
in MutationTsNs, backfilled from TsNs for payloads written before the
field existed. The notification callback forwards it to the wire message.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* pb: normalize chunk-replay timestamps like eachLogEntryFn
ReadLogFileRefs bypasses the server decode path, so a delayed append
still arrived under its stale payload stamp and could hide behind a
subscriber cursor. Deliver the stream position and preserve the stamp in
MutationTsNs.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* filer: order replay and tombstones by the mutation stamp
Replay treated the delivered TsNs as the commit time, so a peer write
delayed in its own log could overwrite a newer local entry. Compare and
stamp with MutationTsNs, falling back to TsNs for older peers.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* remote.sync: carry the mutation stamp through reshaped renames
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* filer: test replay ordering on the mutation stamp
* filer: tombstone remote deletes by delivery position, order by commit stamp
* filer: keep the peer commit stamp when restamping aggregated events
* pb: keep the log cursor on the entry position past callback restamps
---------
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
CI installs the templ CLI with `@latest`, which now generates code
calling templruntime.NewAttributeExpression — a symbol added after the
v0.3.1070 runtime pinned in go.mod, breaking `make install` builds on
clean runners. Bump the runtime to v0.3.1100 and regenerate.
* filer: format ChunkView without its cipher key
readChunkSliceAt logs a failed chunk fetch with %+v, which printed the
per-chunk AES-256-GCM key into error-level logs on every failed read.
Give ChunkView a String() that omits CipherKey so %v and %+v stay safe.
* pb: scrub cipher keys when an entry must be logged whole
ScrubCipherKeys clones an Entry with chunk CipherKey fields cleared, so
a warning that needs the full entry still cannot leak the keys.
* log entry identifiers instead of whole metadata messages
Several log lines formatted whole protobuf messages — SubscribeMetadataResponse,
EventNotification, Entry-carrying requests — whose chunks include cipher_key
under -encryptVolumeData, so the keys escaped into logs (some at default
verbosity). Log the directory, entry name and event timestamp instead; where a
chunk list was printed, print file ids via FileIdsString.
* log queue: unwrap SubscribeMetadataResponse and never print raw protos
The fallback printed any message with %+v, so a SubscribeMetadataResponse
carrying entries would still put cipher keys in the log. Unwrap its event
notifications (batched events included) and reduce unknown messages to
their type name.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* remote sync logs: name the event's real path
Rename targets logged under resp.Directory while the entry lives under
NewParentPath; route them through MetadataEventTargetDirectory. Sync
failure logs printed only the directory, so same-dir failures shared an
identifier — log the source and destination paths instead.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* log queue: log the entry's destination path on moves
A cross-directory rename printed "file" -> "file" under the source
path, hiding where the entry landed. Use NewParentPath so the log
shows the destination directory next to the new name.
* admin: install the go.mod-pinned templ, not @latest
install-templ fetched templ@latest, which now emits
templruntime.NewAttributeExpression - a symbol absent from the
runtime pinned in go.mod - so every fresh 'make' regenerated
uncompilable _templ.go files. Install the go.mod version and
reinstall when the binary on PATH reports a different one.
* admin: drop stray Makefile change belonging to the templ pin PR
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* notification: join the move destination path in log queue output
A NewParentPath of "/" printed //name for moves into the root;
path.Join keeps a single leading slash.
* log queue: skip a nil event notification instead of logging an empty move
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* pb: tolerate nil chunk entries while scrubbing cipher keys for logs
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* subscription and backup failure logs name the event's file
The process/marshal failure logs printed only the event directory and
timestamp, so a failed sync left operators guessing which file failed.
Use MetadataEventSourceFullPath/MetadataEventTargetFullPath so each
failure names the source and target paths without printing the entries.
* replication: identify entry-less events without printing the proto
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* s3: scrub cipher keys from the bucket-metadata debug dump
buildBucketMetadata marshals the whole filer entry to JSON for a V(3)
log line; a chunked entry would emit each chunk's cipher_key base64.
Marshal the scrubbed copy instead.
---------
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* s3: keep transient object-entry errors out of ErrObjectNotFound
getObjectEntry wrapped every fetch error as ErrObjectNotFound, so a
transient filer failure (busy filer, ctx deadline, transport blip) was
indistinguishable from a real miss. LifecycleDelete resolves
ErrObjectNotFound permanently as NOOP_RESOLVED, so one flaky lookup at
dispatch time silently skipped the object's expiration - errors=0, no
retry, no log. This matches the flaky s3tests
test_lifecyclev2_expiration signature: exactly one object of a
same-rule pair surviving past its window.
Only collapse genuine misses (filer not-found, version/latest-version
sentinels); everything else propagates so the worker retries it as
RETRY_LATER instead of resolving it.
Also seed bucketConfigCache in the retention test server and teach the
fake filer to inject per-path lookup errors.
* s3: keep invalid version ids on the not-found path in getObjectEntry
A malformed versionId used to collapse into ErrObjectNotFound like every
other fetch error, so the handlers answered NoSuchKey. With transient
errors now propagated, an invalid id would surface as InternalError;
fold it back into the miss classification.
* filer_pb: keep Unavailable out of LookupEntry's not-found normalization
* s3: reject Unavailable in isObjectEntryMiss before message matching
* s3: keep transient fallback errors out of latest-version resolution
* filer_pb: limit LookupEntry's not-found text match to miss statuses
* s3: limit isFilerNotFound text matching to miss statuses
* vector: project _distance and _rowid explicitly
Lance warns that a knn projection without _distance will stop
auto-including it; spell both scoring columns out.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* vector: reference-comparison tests
The spike's exit gate: assert the engine's answers equal a bare
scan().nearest() on the same pinned version — over a fresh index, over
unindexed fragments, over a two-segment index after optimize_indices,
and after a delete. A version-pinned handle querying through
maintenance sees only its own version.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* vector: weed-vector spike CLI
describe / coverage / query / get against a namespace-registered table
or a plain dataset URI, so the spike runs against a real SeaweedFS Lance
table bucket and against a local fixture alike. Flag and env names
mirror weed-worker.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* ci: rust-vector-tests workflow
Build and test seaweed-vector in release on pushes and PRs touching it
or its path-dependency trees, mirroring rust-worker-tests.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* vector: derive Serialize for the spike CLI's JSON output
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* vector: --postfilter on the spike CLI
Track the Query field rename; the flag stays an explicit opt-out of
prefiltering for comparison runs.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* vector: assert the pinned query's answers
The maintenance test passed even with zero pinned hits: an empty result
satisfies all() trivially. Assert the hit count and compare the pinned
answers against the reference on the pinned handle.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* ci: vector tests also watch plugin.proto
weed-vector-core path-depends on weed-lance-worker, which compiles
weed/pb/plugin.proto through seaweed-worker-core; a proto-only change
would skip these checks and could break this workspace.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* vector: describe prints layout and coverage
The describe doc comment promised the vector layout and index coverage;
print both, and propagate count_rows errors instead of printing 0.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* vector: share the test dataset builder
The coverage tests need the same tempdir Lance datasets the query tests
build; move them into a cfg(test) helper module.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* vector: prefilter the metadata filter
prefilter=false post-filters the top-K rows, which can return fewer than
k results and never sees unindexed rows beyond the nearest cut. Expose
the choice so the S3 Vectors layer can honour AWS filter semantics.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* vector: index coverage introspection
coverage() unions each index's fragment bitmaps against the manifest's
fragment list to report which fragments a vector index covers, which it
has not reached, and how many rows the flat pass still has to scan.
Tests pin the behaviour the spike is for: rows appended after an index
build stay in the top-K, and a version-pinned dataset never sees the
next commit.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* vector: default filtered queries to prefilter
postfilter=false is now the normal path: the metadata filter cuts the
eligible rows before ranking, so a query returns k hits whenever k
eligible rows exist. Lance's post-filter remains reachable through
Query::postfilter for comparison runs.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* vector: intersect index bitmaps with the live manifest
A bitmap records fragments the index covered when built; a fragment the
manifest dropped afterwards stayed in the reported coverage.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* vector: keep bitmap-less indices in coverage
An index whose metadata has no fragment bitmap was dropped from the
report entirely, indistinguishable from no index at all. Count its
segments and mark its coverage unknown instead.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* vector: count live rows in unindexed coverage
physical_rows counts deleted rows and can be absent in the manifest;
sum FileFragment::count_rows for the unindexed fragments instead.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* vector: error on a null key in a selected key column
* vector: compare filtered ANN and flat hits as sets
* vector: coverage reports the checked-out version
* vector: project _distance and _rowid explicitly
---------
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: chrislusf <chris@seaweedfs.com>
* vector: scaffold the seaweed-vector workspace
Standalone cargo workspace for the vector query engine, mirroring the
seaweed-worker layout: own lockfile, same lint policy, path dependencies
across the two trees since there is no common root manifest.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* vector: resolve a table's vector layout
VectorTable::resolve finds the vector column (explicit name, else the
column named vector, else the sole fixed-size float list), the optional
key column, and the metadata columns a query can project.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* vector: flat and indexed top-k, key lookup
query::flat runs an exact scan (use_index(false)); query::ann lets Lance
split the query between index segments and a flat pass over fragments no
index covers yet, so rows written after the last index build stay
eligible. Both honour a metadata filter and return the raw _distance in
the query's metric.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* vector: reject a non-utf8 or non-identifier key column
An explicit --key-column naming a non-UTF8 field made hits() silently
report row ids as keys, and get() interpolated the name into a SQL
filter verbatim. Require Utf8 at resolve and in hits(), and require a
plain identifier before building the IN predicate.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* vector: fall back to row id on a null key
keys.value() on a null slot fabricates an empty-string key; a row with
no key reports its row id instead, matching the no-key-column case.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* vector: quote the key column in the IN filter
A Utf8 key column named e.g. document-key made the predicate parse as
subtraction, and an identifier interpolated unquoted is injectable.
Backtick-quote it and double its inner backticks; double quotes are
string literals in the filter dialect, not identifiers.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* vector: project only the columns a hit needs
The knn projection pulled every metadata column — including other
vector columns — although Hit returns none of them. Filters still
evaluate off the scan path, so restrict the output to the key and the
scoring columns.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* admin: install the go.mod-pinned templ, not @latest
install-templ fetched templ@latest, which now emits
templruntime.NewAttributeExpression - a symbol absent from the
runtime pinned in go.mod - so every fresh 'make' regenerated
uncompilable _templ.go files. Install the go.mod version and
reinstall when the binary on PATH reports a different one.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* admin: invoke templ from Go's install dir, not PATH
A templ binary earlier on PATH than GOBIN still shadowed the pinned
compiler after install-templ ran, so generate kept emitting code for
the wrong runtime. Resolve GOBIN (or GOPATH/bin) and call the pinned
binary by absolute path for install checks, generate, watch, and fmt.
* admin: regenerate templ output with the pinned v0.3.1070
The checked-in _templ.go files were stamped v0.3.1020, older than the
go.mod pin. Regenerate so the committed code matches the compiler the
runtime links against.
* admin: resolve templ under the first GOPATH entry
go install drops binaries in the first GOPATH element's bin, so a
multi-element GOPATH left TEMPL pointing at a path that does not exist.
Split on the OS list separator so Windows GOPATHs resolve too.
* admin: install pinned templ before fmt and pattern-rule generates
fmt and a direct %_templ.go rebuild invoked TEMPL without ensuring the
pinned binary exists; an order-only prerequisite keeps the check cheap
without forcing regeneration.
* admin: keep spaced GOPATH/templ paths intact in the Makefile
---------
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* master, volume, filer: start on a read-only mounted directory and serve existing data read-only
A read-only mount was treated as writable (permission bits only), so every
service failed on its first write: the master panicked writing raft conf.tmp,
the filer exited opening the leveldb LOCK, and the volume server skipped every
volume while still offering the directory as free capacity.
- util.IsReadOnlyMount: access(W_OK) == EROFS.
- volume: on a read-only -dir take the existing O_RDONLY/noWriteOrDelete load,
index .idx in memory when no .sdx exists, offer no free slots (same branches
as isDiskSpaceLow), skip the vid stamp there.
- master: on a read-only -mdir run raft from a private copy and refuse new
volume ids.
- filer: leveldb2/leveldb3 set their existing ReadOnly option.
* volume: skip read-only mounts when picking EC shard copy targets
A shard copy landing on a read-only mount fails at write time; offer
only writable locations like the other placement paths do.
* volume: mirror the read-only mount handling in the Rust volume server
The .dat open only fell back to read-only on EACCES, so a real read-only
mount (EROFS) still failed the load. The index side already degrades
once no_write_or_delete is set. Add is_read_only_mount via access(W_OK)
== EROFS, mark DiskLocation.read_only_mount, and mirror the Go checks:
no free slots, heartbeat max = used, no vid-stamp writes, no EC shard
copy targets, and a read-only index dir forces the volume read-only.
* volume: write vid stamps while only the index dir is read-only
* filer: skip the write-bit check when a leveldb store opens read-only
* filer: serve reads of buckets with no leveldb3 dir on a read-only store
* volume: derive a stable dir uuid when a read-only mount cannot store one
* master: keep the read-only raft copy across restarts so votes survive
* volume: index a missing .idx in memory on a read-only mount
* volume: release the .idx handle after an in-memory read-only load
* volume: mount EC shards read-only when their index dir is on a read-only mount
* volume: mark the index dir read-only before the tail repair opens .idx
* volume: write vid stamps while only the index dir is read-only (Rust)
* volume: derive a stable dir uuid and report no free slots on a read-only mount (Rust)
* volume: index a missing .idx in memory on a read-only mount (Rust)
* volume: refuse an EC copy whose shard owner is on a read-only disk
* master: key the read-only raft copy by the absolute -mdir
Two masters sharing the same relative -mdir text from different working
directories would have collided on one temp copy and shared raft state.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* volume: mix device and host into the read-only dir uuid
Deriving the uuid from the path alone let independent servers at the
same mount point report one uuid, which RegisterUuids rejects. Hash in
the device id and hostname so restarts stay stable but disks differ.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: vic6996-web <320442181+vic6996-web@users.noreply.github.com>
Co-authored-by: user.email <chrislusf@users.noreply.github.com>
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix 12074
* volume: shuffle a copy of cached replica locations for proxy reads
LookupVolumeId serves cached location slices shared between lookups; shuffling in place races with concurrent proxied reads of the same volume.
* seaweed-volume: forward inbound method and surface lookup failure in proxy reads
Mirror the Go handler: proxy_request used to always send GET, turning a client HEAD into a replica download, and a failed master lookup returned 404 like missing data. Forward the inbound method and answer 500 for lookup failures in proxy mode.
* volume: keep master lookup error detail out of the proxy error response
* volume: bound the proxied-request wait in the method-preservation test
* seaweed-volume: answer 404 for missing volumes in proxy-mode lookups
Mirror Go's LookupVolumeId: a per-volume error in the master's lookup
reply (e.g. a missing volume) means "no locations" and answers 404
downstream, not a lookup failure that proxy mode answers 500.
* volume: guard nil lookup result in proxy path
A master response that omits the requested vid leaves
LookupVolumeId returning a nil *LookupResult with a nil error;
dereferencing it panics the read handler.
* seaweed-volume: answered lookup errors are missing data, not failures
A master reply carrying a per-volume error with HTTP 404 (volume not
found) now yields empty locations like Go's LookupResult, so proxy mode
answers 404. Other failure statuses such as 503 during master warmup
remain lookup failures that proxy mode answers 500.
* seaweed-volume: drop hop-by-hop headers from replica proxy reads
Go's inbound request drops Host, Content-Length and Transfer-Encoding
into dedicated fields, so its forwarded header set never carries them.
hyper keeps them; forwarding a client's Content-Length with no body
stalls the replica's read, and forwarding its Host misnames the target.
* volume: bound the proxy master lookup to the request context
* volume: drop hop-by-hop and Connection-nominated headers from proxy reads
* seaweed-volume: drop Connection-nominated headers from proxy reads
* volume: skip the proxy lookup error when the inbound request is gone
* volume: drop proxy-specific headers from replica proxy reads
* seaweed-volume: drop proxy-specific headers from replica proxy reads
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* volume: drop hop-by-hop headers from replica proxy responses
* seaweed-volume: drop hop-by-hop headers from replica proxy responses
* seaweed-volume: a failed lookup status without an error body is still a failure
* volume: do not re-proxy an already-proxied read for a missing volume
A proxied request that reaches a server which still lacks the volume was
looked up and forwarded again, so a stale location answer could bounce
the read between two servers until the client gave up. The Rust volume
server's proxy_missing_volume already returns 404 for proxied=true;
mirror it here.
* seaweed-volume: replace stale proxied params in the forwarded query
Go builds the proxied URL with query.Set(reqIsProxied), which replaces
every proxied param the client sent. Appending after a client-supplied
proxied=false left the replica's first-value lookup reading the stale
marker, defeating the loop guard.
---------
Co-authored-by: chrislusf <chrislusf@users.noreply.github.com>
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* pb: checkpoint metadata-log replay progress per settled entry
A chunk-mode subscriber's resume cursor only advanced after the whole
LogFileRefs batch drained, so a stream dropped mid-replay resubscribed
from the original stale position and re-read the same log files while
live traffic kept appending - under a flapping filer the replay window
never shrank, and each reconnect re-listed and re-fetched the same
backlog.
ReadLogFileRefs now reports every entry the merge has settled -
delivered or filter-skipped - and FollowMetadata checkpoints StartTsNs
on it, so a reconnect resumes where the previous stream stopped and the
backlog drains across restarts. The peer aggregator wires the same
progress into its resubscribe/offset position.
* pb: emit a replay checkpoint only after the merge passes its timestamp
Checkpointing an entry's timestamp the moment it settles could strand a
same-timestamp entry still queued on another filer's stream: resuming is
exclusive, so a reconnect would skip the unsettled half of the tie. The
merge now emits a timestamp only once a later entry proves every tied
entry was delivered (and once at clean EOF).
* pb+filer: never regress the replay cursor below the settled checkpoint
The drain's final assignment took the last settled entry's own timestamp,
which can lag the log position progressFn just checkpointed - a skewed or
legacy entry rewinds option.StartTsNs (and the peer's lastTsNs), so the next
drain or a resubscribe re-reads the settled prefix. Guard both assignments
so the cursor only moves forward.
* pb: cover the resubscribe path of replay checkpoints
A drain that aborts mid-replay leaves StartTsNs at the last checkpoint
the merge passed, and the next subscribe resumes there instead of
re-reading the whole ref set. The test drives the real
makeSubscribeMetadataFunc path: refs, a mid-batch apply failure, then a
resubscribe whose SinceNs must carry the checkpoint.
* pb: emit the tail replay checkpoint only for a settled timestamp
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(volume): keep preallocated growth above -volume.minFreeSpace
When free space fell below -volume.minFreeSpace, every volume went
read-only and the master kept growing new volumes. With preallocation on,
each new volume reserved its full size, driving the disk further below the
floor (15 Oct incident: 66 empty volumes reserved about 1980 GiB).
Two causes, both in weed/storage/store.go:
- MaybeAdjustVolumeMax sized a disk's volume slots from raw free space, so
space inside the floor was counted as available. It now counts only free
space above the floor.
- addVolume trusted the once-a-minute low-disk flag. A preallocated volume
is now refused when the disk is low now, or when its reservation would
not fit above the floor. Volumes without preallocation, including merges
that copy existing data, are not affected.
addVolume runs under a store-level lock, so two concurrent creates cannot
both pass the room check against the same free space.
* volume: mirror headroom guard in the rust store
find_free_location only trusted the once-a-probe is_disk_space_low flag,
so a burst of preallocated creates could still run the disk under
-volume.minFreeSpace between probes, and maybe_adjust_volume_max sized
slots from raw free space. Adds MinFreeSpace::is_low/available_space to
match Go and applies both checks the same way. store.write() already
serializes add_volume, so no extra lock is needed.
* volume: read rust headroom tests from a fake disk
The floor arithmetic used a live free-space reading, so unrelated disk
activity could flip a test between pass and fail. Store.disk_stats_of is
now the same seam Go uses (diskStatusOf): the growth-loop and floor tests
run against a controlled disk, mirroring the Go suite one for one.
* volume: hold back create metadata from the headroom check
An exact-fit preallocation left free space exactly at the floor, so the
.vif and .idx writes that follow a create could dip under it and get the
location flagged read-only at the next probe. Reserve room for .vif on
the data filesystem, and for .idx growth only when dir_idx shares it.
* volume: mirror the metadata allowance in the rust store
Same rule as the Go side: charge .vif against the checked filesystem and
.idx only when dir_idx shares it, and grow the exact-fit test's fake
disk by the per-create cost.
* volume: skip the reservation check where create cannot preallocate
* volume: mirror the platform preallocation gate in the rust store
* volume.copy: count the preallocation reservation in the destination space check
The copy path picked a location with room for the copied .dat, then reserved
the master's full volume size limit on it. A volume smaller than the limit
could land on a disk whose free space above -volume.minFreeSpace fit the
copy but not the reservation, driving the disk under the floor the same way
the growth loop did.
Fetch the preallocation setting before choosing the location and require
room for the reservation itself, only where CreateVolumeFile actually
reserves blocks.
* seaweed-volume: store min-free-space-adjusted available space like Go
Go's DiskLocation.AvailableSpace holds free space above the
-volume.minFreeSpace floor; the Rust probe cached raw free instead, so the
VolumeCopy location check could accept a disk already inside the floor.
Store the same adjusted figure and cover it with a test.
* seaweed-volume: reserve the copied .dat like Go's CreateVolumeFile
VolumeCopy called set_len for the reservation, which only grows a sparse
file — no blocks are held, so a disk with room for the copied bytes but
not the full reservation only fails mid-copy. preallocate_file fallocates
KEEP_SIZE on Linux like Go, reserving the blocks up front; off Linux it
is a no-op, matching PREALLOCATE_SUPPORTED.
* seaweed-volume: charge a copy's destination pick for the .dat reservation
Go's VolumeCopy now sizes its free-location check against the master's
volume size limit when the platform preallocates, not just the copied
bytes. The Rust pick only measured the copied bytes, so a destination
could accept a copy whose reservation then dips under the floor. The
config fetch moves ahead of the pick (same order as Go) and feeds both
the space check and the reservation itself.
* volume.copy: charge the picked location's free space under volumeCreateLock
The copy path picked a destination off the once-a-minute AvailableSpace
probe and released the decision long before the .dat reservation landed.
Two copies could pass the same cached headroom and together drive the disk
under -volume.minFreeSpace.
ReserveFreeLocation serializes the pick under volumeCreateLock, verifies
the disk still has live room for the reservation, and charges it against
the location's cached free space until the next probe refreshes it.
* seaweed-volume: mirror the copy pick's live-check and headroom charge
Go's VolumeCopy now picks through Store.ReserveFreeLocation: live disk
stats under the store write lock, then the reservation charged against the
location's cached free space. The Rust plan_copy_destination gains the
same store.reserve_free_location so two copies cannot spend the same
headroom before the first .dat reservation lands.
* volume.copy: saturate the free-space charge against a live-space drop
The two statfs reads bracketing a pick can race a large external write; a
plain subtraction would underflow and leave the location looking roomy
until the next probe.
* volume.copy: release the headroom reservation when a copy does not land
ReserveFreeLocation charges the picked location's cached free space, but
a failed copy never returned it, so the headroom stayed booked until the
next disk-space probe and blocked retries. ReleaseFreeLocation refunds
the charge; the copy's cleanup defer now registers with the reservation
so every failure path (existing-replica delete, .note write, stream send)
releases it, including the post-transfer mount/validate failures covered
by the outer defer.
* seaweed-volume: mirror the copy headroom release on failed copies
release_free_location refunds reserve_free_location's charge. The copy
returns it when the plan's existing-replica delete fails, when the .note
write fails, and in cleanup_failed_copy, so a failed copy stops holding
the headroom until the next disk-space probe.
* volume.copy: keep pending copy headroom out of probe readings
ReleaseFreeLocation refunded the reservation into AvailableSpace even when
a CheckDiskSpace probe had already rebased the cache on live stats — the
probe's reading still shows the in-flight copy's bytes as free, so the
refund double-counted headroom and two copies could spend the same space,
driving the disk under -volume.minFreeSpace.
Track the outstanding charge in DiskLocation.ReservedSpace instead: probes
subtract it so they cannot re-spend it, a failed copy removes only its own
charge, and a landed copy commits it — the charge and the new usage cancel
out. hasRoomForNewVolume respects pending headroom too, so a preallocating
create cannot overrun an in-flight copy.
* seaweed-volume: mirror pending copy headroom across disk probes
reserved_space tracks the in-flight copy's charge separately from the
cached free space: check_disk_space subtracts it, release removes only the
failed copy's charge, a landed copy commits it, and has_room_for_new_volume
respects it — matching the Go side.
* volume.copy: settle a copy's headroom as its bytes land on disk
The pending charge held the copy's full reservation until commit, but
written and fallocated bytes already show up in statfs, so the live-space
check counted them twice. A slow copy could then block other copies and
volume growth despite free space above -volume.minFreeSpace.
Settle the charge instead: commit the bytes CreateVolumeFile actually
reserved (a failed fallocate leaves nothing committed), then settle each
file's write progress as it lands. The failure paths now release only the
unsettled remainder; allocatedSize is exported for the .dat's st_blocks.
* seaweed-volume: mirror copy headroom settling as bytes land
reserved_space keeps only the promised bytes statfs cannot see yet:
preallocate_dat settles whatever blocks the filesystem actually reserved
(0 when fallocate cannot), each copied file's progress settles the rest as
it lands, and a failed copy releases only the unsettled remainder —
matching the Go side.
* volume.copy: release a failed copy's headroom charge only once
Both defers fired on the same in-closure error: the inner released the
unsettled charge and the outer released it again, driving ReservedSpace
negative (an atomic.Uint64 underflow) and inflating AvailableSpace —
the location would then fail every later headroom check, or worse,
pass them with fabricated space. Zero the charge once it is released;
the Rust mirror already releases via swap(0) for the same reason.
* volume.copy: write into the preallocated .dat instead of staging over it
The .dat stream staged into .copying and renamed over the file
CreateVolumeFile had just fallocated, so the copy's transient footprint
was preallocate+dat — more than the reservation the pick just checked —
and the reservation itself was discarded at rename. Writing in place
reuses the fallocated extent, keeps the peak at the reserved maximum,
and leaves the copied volume preallocated like a created one.
* seaweed-volume: mirror the in-place write into the preallocated .dat
The .dat copy opened the just-fallocated file with truncate, freeing its
blocks at open and spending the headroom again as the stream re-grew it.
Append mode preserves them, matching the Go side.
---------
Co-authored-by: chrislusf <chrislusf@gmail.com>
Co-authored-by: user.email <chrislusf@users.noreply.github.com>
* fix: volume.fsck -volumeId purges live chunks referenced from outside the volume's bucket
-volumeId narrowed the filer read to <bucketsPath>/<collection> when
the selected volumes shared one bucketed collection (#9347). A filer
started with -collection, or an fs.configure rule, stores files from
any path in that collection, so the narrowed read misses their
references and reports live chunks as orphans; with
-reallyDeleteFromVolume they are deleted. #11794 already made
-collection read from the root for this reason. Do the same for
-volumeId: the flag keeps selecting which volumes are checked and
purged, and the filer is always read from "/".
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
* test: volume.fsck keeps chunks referenced outside the volume's bucket
---------
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
Co-authored-by: user.email <chrislusf@users.noreply.github.com>
* iceberg: fix CREATE TABLE rejecting a partition on the first column
Apache Spark number the columns of a new
table from 0 and leave assigning fresh IDs to the catalog, so a table
partitioned or sorted by its first column arrives with source-id 0. The
create-table request decoded its partition spec and write order into
iceberg-go's bound PartitionSpec and SortOrder, which require positive
source IDs, so every such CREATE TABLE, and CTAS through stage-create,
was refused with "Malformed request: Invalid request body".
Decode both as iceberg.UnboundPartitionSpec and table.UnboundSortOrder,
as iceberg-go documents for catalog create-table requests;
newTableMetadata already resolves the placeholders through the request
schema by name and assigns fresh IDs. A spec or order that does not
resolve against the schema is now answered with 400 instead of 500, and
the decode error message carries the underlying reason.
Tests check that Spark's request bodies resolve to the same metadata as
1-based placeholders, that the handler accepts them, that unresolvable
source IDs get 400, and add a Spark integration test that partitions by
the first column.
* iceberg: gofmt the create-table source-id tests
---------
Co-authored-by: user.email <chrislusf@users.noreply.github.com>
* filer: support nanosecond timestamps for conditional delete (#11339)
* shell: fs.verify prune carries the entry mtime in nanoseconds
IfNotModifiedAfter now resolves to sub-second precision on the filer,
so pass the looked-up mtime_ns along with mtime; the re-lookup check
catches same-second rewrites that share the second, and the test mock
mirrors the filer's seconds/nanoseconds cutoff.
* shell: volume.fsck guards empty-directory deletes with mtime nanoseconds
The looked-up mtime_ns joins the seconds in IfNotModifiedAfter so a
directory rewritten within the same second is not dropped; the stub
filer mirrors the filer's scale heuristic and the race fixtures use a
realistic mtime so the nanosecond branch engages.
* filer: cover the seconds guard and equality boundary
Test second-resolution cutoffs on both sides of the entry's second and
the exact-equality nanosecond guard that still deletes.
* filer: compare the nanosecond guard as a time, not UnixNano
UnixNano overflows for mtimes outside 1678-2262, letting a far-future
entry slip past the guard; time.After compares exactly at every epoch.
The seconds branch keeps Unix() so a same-second cutoff still deletes.
* filer_pb: add explicit if_not_modified_after_ns to DeleteEntryRequest
* filer: prefer explicit nanoseconds guard, keep if_not_modified_after as seconds
* shell: send seconds and nanoseconds delete cutoffs, skip unexpressible mtimes
* filer: merge conditional-delete guards into one nanosecond cutoff
DeleteEntryMetaAndData carried both the seconds and nanoseconds guards
even though the precedence between them is fixed, so every caller passed
a dead parameter. The gRPC handler now folds if_not_modified_after into
nanoseconds at the boundary, where (sec+1)*1e9-1 keeps the whole-second
cutoff exact, and the filer takes a single ifNotModifiedAfterNs.
* filer: hold the conditional-delete cutoff as a time, not int64 nanoseconds
A seconds-only guard at or past year 2262 saturated to MaxInt64
nanoseconds (~2262), so a delete whose cutoff matched a far-future mtime
was skipped while the request still reported success, and any entry
between 2262 and the real cutoff survived pruning. The merged cutoff is
now a time.Time, where a whole-second guard maps to the last nanosecond
of its second for any int64 second.
---------
Co-authored-by: user.email <chrislusf@users.noreply.github.com>
Co-authored-by: Chris Lu <chris.lu@gmail.com>
* s3: group ListMultipartUploads by delimiter into CommonPrefixes
The delimiter was echoed but ignored, so every upload was listed. Keys
that continue past the delimiter after the prefix are now listed once as
a common prefix, counted toward max-uploads like an upload.
* s3: bound tracked multipart common prefixes to the page
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* s3: omit upload id marker when a page ends on a common prefix
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: user.email <chrislusf@users.noreply.github.com>
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* webdav: copy a file onto a collection URL without losing the collection
An overwriting COPY of a file to dir/ removed dir, then OpenFile refused
the name ending in / and the request failed with 403, leaving neither.
Drop the trailing slash from the destination when the source is a file.
* webdav: cover file-over-collection COPY through the real handler
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* webdav: keep a root destination out of the slash trim
A file COPY or MOVE onto / would rewrite the Destination to an empty
path, turning the existing refusal into x/net's bad-gateway error; leave
the root untouched so RemoveAll refuses it as before.
---------
Co-authored-by: user.email <chrislusf@users.noreply.github.com>
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
The 24.04 bump broke build-rust-binaries: aws-lc-sys C objects compile against noble's glibc 2.39 headers, which redirect sscanf/strtol to the C23 __isoc23_* variants musl does not provide, failing the link.
Same fix as the enterprise build on seaweedfs/artifactory:
- arm64: keep aarch64-linux-gnu-gcc, link a tiny forwarding shim that
resolves __isoc23_sscanf/__isoc23_strtol to musl's vsscanf/strtol.
- amd64: point CC at musl-gcc so C deps compile against musl headers.
* iam/oidc: drop single-use web-identity tokens and the jti replay cache
Reverts the single-use semantics added in #11936 (and the #12017
refactor): AssumeRoleWithWebIdentity consumed each web-identity
token's jti into a bounded replay cache, so a second exchange of the
same token was rejected until exp + skew.
AWS STS accepts a web-identity token repeatedly until it expires,
and Kubernetes workload identity depends on that: the kubelet keeps
a projected ServiceAccount token until ~80% of its lifetime, so pod
restarts, co-located containers sharing the token file, repeated CLI
invocations, and pre-rotation SDK refreshes all re-present the same
jti and were refused on 4.50.
The replay machinery also carried a real maintenance cost — a
bounded concurrent cache, fingerprint-keyed entries for jti-less
tokens, adopt/merge handoffs so provider rebuilds could not reopen
consumed tokens, and provider-reuse bookkeeping in the refresh path
that existed only to preserve that state. Removing it restores AWS
semantics and deletes the whole apparatus.
Refs #12063. Alternative to the per-provider opt-out in #12064.
* iam: reuse unchanged OIDC providers on refresh
refreshOIDCProvidersFrom rebuilt every managed provider on any store
change, so an unrelated write discarded each provider's JWKS cache and
an unreachable issuer could then fail exchanges the cached keys would
have allowed. Keep a per-record provider cache keyed on the record
content fingerprint and only rebuild when the record changes.
* iam/oidc: require exp in ValidateToken
Dropping the single-use check also removed the implicit bound on
expiry-less tokens: a signed web-identity token without exp could
mint STS sessions indefinitely. Require the exp claim, matching AWS
AssumeRoleWithWebIdentity which rejects tokens without one.
* iam/oidc: bound idle connections on the provider transport
A provider replaced on refresh drops its http.Transport without a
close; give it an IdleConnTimeout so pooled connections and their
goroutines cannot linger.
* s3api: cover web-identity token reuse over HTTP
Sign one token and exchange that exact string twice, directly covering
the Kubernetes projected-token reuse this change restores.
* test(iam): refresh reuses an unchanged record's provider
Covers the provider-reuse cache: an unrelated record's refresh keeps a
provider's JWKS state, a changed record rebuilds it.
* s3: answer GET and HEAD of a delete marker as S3 does
A delete marker as the current version now returns 404 NoSuchKey with
x-amz-delete-marker: true and the marker's x-amz-version-id. Asking for the
marker by versionId returns 405 MethodNotAllowed with the same headers and
Last-Modified, instead of 404 NoSuchKey.
* s3: check the tag-conditioned read grant before answering a delete marker
The marker branch returned before the entry-based policy recheck the rest
of the read path performs, so an s3:ExistingObjectTag-conditioned Deny did
not apply to the delete-marker response and its version headers.
* s3: send Allow: DELETE with the delete-marker 405
RFC 9110 15.5.6 requires a 405 answer to list the methods the resource
supports; S3 sends allow: DELETE for a delete marker named by versionId.
* s3: keep delete-marker headers on conditional-read misses
If-Match or If-Unmodified-Since against a current delete marker short-
circuited to a bare NoSuchKey. Carry the resolved marker through the
conditional result so the 404 keeps x-amz-delete-marker and
x-amz-version-id, matching the unconditional read.
* s3: exercise delete-marker responses through GET and HEAD handlers
* s3: read marker version id from the entry before the caller's guess
A delete marker written outside the normal PUT path can miss the
version-id attribute while the v_<id> file name still names it; the
new x-amz-version-id header would then report "null". Reuse
versionIdFromEntry so both the handler and conditional-miss paths
emit the real id.
* s3: answer 405 for versionId=null on a suspended-versioning marker
The null marker lives in .versions while getSpecificObjectVersion only
checks the plain path, so GET/HEAD ?versionId=null answered
NoSuchVersion. On that miss, look up the null marker and answer it
like any other marker named by versionId.
* s3: propagate null-marker lookup failures instead of answering 404
A transient filer error reading .versions/v_null was swallowed as
"no marker", so GET/HEAD answered NoSuchVersion for a live read
failure. NotFound and absent entries still mean no marker; anything
else is InternalError now.
---------
Co-authored-by: Chris Lu <chris.lu@gmail.com>
Co-authored-by: chrislusf <chrislusf@users.noreply.github.com>
* filer: enable per-path commit records only when a peer has its own store
Since seaweedfs#11698 every metadata change writes a per-path commit
record into the filer store (an MpTs key, plus an MpDl key on deletes),
and the records stay after the path is deleted. They order the events a
filer replays from peer filers that keep their own store, since replay
only runs when peerSignature != f.Signature.
Filers that all share one store (one Redis, MySQL, TiKV, ...) never
replay each other's events, so the records are never read, yet each
change still costs one or two extra store writes and the key count grows
with every path ever changed — unbounded growth for in-memory stores.
This turns the records off by default and latches them on the first time
a peer reports a different store signature — no option needed. The latch
is persisted in the store (MpRep) so it survives restarts and converges
filers sharing a store even before they discover the peer themselves.
While off, a filer keeps an MpOff marker so enabling later bumps the
records' generation (MpGen, folded into each record key) past the
unrecorded window: a record older than a change made while off cannot
let an older replayed peer event win. Ordering for unrecorded changes
falls back to the entry's mtime, the same as entries written before the
records existed.
Supersedes the manual option approach in seaweedfs#12060.
* filer: keep delete tombstones while records off, harden enable path
Review on the first commit:
- A delete committed while the records were off left no entry and no
usable tombstone: after enabling, an older replayed create could
resurrect the path. Deletes and rename sources are now always recorded
under the generation-0 MpDl key — they are never fenced by a
generation bump, and they cannot go stale because every delete is
recorded. Creates/updates still write nothing while off.
- bumpPathCommitGen could regress MpGen under concurrent bumpers on a
shared store. It now loops: each pass writes only a generation higher
than any it has seen, so concurrent bumps converge upward.
- doSubscribeToOneFiler proceeded into replay when
enablePathCommitRecords failed, running the whole session without
ordering. It now aborts the subscribe so the reconnect retries with
the records on.
* filer: bind pending commit-record repairs to their generation
A failed MpTs write kept in metaTsPending could land after a shared-store
generation bump, writing a pre-bump timestamp as a current-generation
record and overriding newer unrecorded state. Each pending change record
now carries the generation it failed under: repairs write it only while
that generation is current, and ordering reads ignore it otherwise.
Pending delete records need no binding — MpDl keys are never fenced.
* filer: never move the local records' generation backward
The converge loop in bumpPathCommitGen seeded from zero and both it and
syncPathCommitGen stored unconditionally: a stale writer could leave the
stored generation below what this filer already adopted, and the next
sync or bump then walked the local value back — orphaning records it
wrote under the higher generation. The bump now seeds from the local
generation and every store only moves it forward.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* filer: name a records generation after the off marker it consumes
Two filers consuming the same off marker each counted one past the
stored generation, so the second could store N+2 after the first
stored N+1 and write records under a key the other could not read
until the next check. The bump now takes the marker's own timestamp —
filers racing on one marker land on one value — a marker older than
the stored generation is cleaned as a leftover, and a stored value
that already passed the bump is adopted instead of counted past. A
newly enabling filer whose marker was already consumed names its own
off window with the current time, so its unrecorded period is still
fenced.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* filer: tell a fresh off marker from one the generation already consumed
A marker whose timestamp reads at or below the stored generation was
dropped as leftover, but a new off period can also read that old — the
off filer's clock lagging, or a host clock moving back — and dropping
it kept records from before that period live, so stale replay could
win.
The stored generation now names the marker it was bumped for: a marker
the store still shows is consumed unless it is that one, however old
it reads, and only the consumed marker itself is cleared without a
bump. The generation trails in the value, so a whole-slice read still
yields it on stores written before the pair was stored. Enabling
fences the filer's own off window with the current time instead of a
marker a sharer may already have consumed.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* filer: cover racing off-marker consumers settling on one generation
* filer: adopt the stored generation, not the consumed marker it names
A filer syncing with no off marker present read the value's first field —
the consumed marker's timestamp — so a restart after a lagging-clock off
period under-counted the generation and kept records the bump dropped.
* filer: re-check the off marker before consuming it
A sync whose marker read ran behind a later off period's consume cycle
still bumped for its stale read: the stored generation already covered a
different marker, so the bump overwrote the consumed name, counted the
generation past records a newer consumption meant to keep, and cleared
whatever marker was live by then. Each pass now re-reads the marker and
only bumps or clears while it is still the one the pass saw.
* filer: keep a failed generation read from resetting the records' generation
The final generation read ignored errors and then stored whatever gen
held — possibly zero — so a transient store failure could hide records
the current generation wrote. Unexpected read errors now return before
the local generation changes.
Also cover a sync whose marker read runs behind another consumer's
delete: it must adopt the stored generation, not bump for the stale read.
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* filer: fail a bump that cannot verify the stored generation
A generation read that failed mid-bump was treated as absent: the
write could land below a generation a racing bumper had stored, or
re-store a marker that bumper had already consumed. Read failures
now abort the pass like every other store error on the path.
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* filer: cover generation reads that fail instead of returning a value
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* filer: carry the commit timestamp on the entry itself
The per-path MpTs record only exists because a commit timestamp had
nowhere else to live. Stamping commit_ts_ns on the entry before its
store write - and reusing that stamp as the emitted event's TsNs -
makes the entry its own commit record: replay and bootstrap traversal
order on it with zero extra writes for creates and updates.
Deletes still need a durable key since a removed entry cannot carry a
timestamp, so MpDl tombstones stay (one write per delete, growth
bounded by delete history). Entries predating the field fall back to
mtime, matching what peers and old records already assume.
This replaces the MpTs/MpOff/MpGen/MpRep machinery entirely: shared-
store filers write nothing extra, and dedicated-store replication
keeps last-writer-wins ordering plus tombstone coverage for deletes
and ancestor drops.
* filer: keep a directory update replay from tombstoning its subtree
A same-path update wrote a delete record before the delete-and-reinsert,
so a directory metadata change read as a subtree delete: a later peer
event for a child older than the update was then skipped even though
nobody deleted the children, leaving files missing on this filer. Only
file updates take the tombstone now — the re-inserted directory entry's
own commit stamp orders the path itself.
* filer: order unstamped entries by their recorded commit before mtime
Entries written before stamping existed have no CommitTsNs, so the
ordering check fell back to the client-settable mtime and dropped the
MpTs record those versions wrote: an entry committed at event time 300
with mtime 100 lost to a delayed peer event at 200 after upgrade. Read
the legacy record for unstamped entries in localCommitTsNs, CommitTsNs,
and the traverse stamp (which also stopped falling back to mtime,
regressing entry_ts_ns to 0).
* filer: release queued rename event stamps when the rename fails
Each moved entry carried an in-flight stamp its queued event would
adopt at emit time. When the move succeeded but a later child move or
the commit failed, the handler rolled back without emitting the queued
events or releasing their stamps, pinning the delivered and flushed
watermarks below the aborted rename forever. Release every queued
stamp on both failure exits in both rename handlers.
* filer: apply a directory update replay with an in-place upsert
Deleting the stored entry before inserting its replacement left a
window where the directory had no entry and no ordering record — a
tombstone cannot stand in for it without reading as a subtree delete
that suppresses the children's own records. A stale peer event accepted
in that gap could replay an older update or delete and sweep the
directory's children. Upsert the replacement instead: the entry keeps
its recorded commit order across the whole update, and a directory has
no chunks or hard links for a delete to release.
* filer: update an existing directory entry on replay instead of upserting
InsertEntry is not an upsert on SQL stores configured without upsert
support — PostgreSQL with enableUpsert=false issues a plain INSERT and
its duplicate-key error does not match the MySQL-only fallback, so a
replay updating a present directory failed on every retry. Pick
UpdateEntry when the entry exists and InsertEntry when it is absent;
UpdateEntry touches no row that is not there.
* filer: detect a shared-store delete racing a replayed directory update
UpdateEntry reports success even when the row was deleted between the
find and the update — the UPDATE changes zero rows and the replayed
change is dropped while the subscription offset advances. Re-read the
entry after the update and retry the write, re-checking the recorded
commit order each pass so a newer peer change wins instead of being
resurrected over.
* filer: declare commit_ts_ns in field-number order
Field 6 sits between extended and hard_link_id; the declaration now
reads in the same order. Wire format unchanged. Also syncs the Java
client's checked-in copy of filer.proto, which the pb Makefile copies
on gen.
* filer: option to skip delete tombstones entirely
filer.options.deleteTombstones=false stops writing MpDl records on
deletes - a single filer or shared-store cluster never replays peer
events, so the tombstones are dead weight. The tradeoff is explicit:
if a dedicated-store peer does replicate, a stale event can resurrect
a deleted path.
* filer: drop delete tombstones entirely
Deletes no longer write MpDl records, and the pending-repair machinery
they needed is gone with them: per-mutation metadata overhead is now
zero for every store topology. Ordering information lives only on the
entry itself via commit_ts_ns.
The accepted tradeoff: a dedicated-store peer replaying an event older
than a delete can resurrect the path, and a kept directory no longer
fences stale child creates. Recorded tombstones and MpTs records left
by older builds are still read, so upgrades keep the protections they
already have.
* filer: stop reading legacy per-path records too
Drops the MpTs/MpDl read path along with the writes: meta_path_ts.go
keeps only the entry-field readers (CommitTsNs, EntryCommitTsNs,
localCommitTsNs). Entries written before stamping existed fall back to
mtime, and tombstones or commit records older builds left behind no
longer fence replayed events.
* filer: trim commit timestamp comments
---------
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* mini: lift the post-mint allocation hold on first inventory report
DeferAllocation armed a fixed warmup window whenever the leader minted a
fresh topology id, so the first write on a brand-new weed mini blocked
for the whole window even though no prior cluster state could exist.
The window still has to cover a rebuilt master's veteran servers, which
report on their first post-reconnect heartbeat.
Mini has exactly one possible reporter — the embedded volume server,
whose first heartbeat carries the full inventory — so hold allocation
only until that report is applied. The bound raise and topology-id
freeze checks run while the heartbeat is processed, so a wiped master
state still raises the bound or trips the freeze before the hold lifts.
Standalone weed master and weed server keep the fixed window: volume
servers outside the process can still be on their way.
* master: apply queued topology-id reports before opening the allocation gate
SetTopologyInitPending(false) cleared the pending flag before replaying
topology ids stashed while the local id was still being minted. A
NextVolumeId landing in the drain window saw no gate and no freeze yet.
The post-mint allocation hold used to cover the gap by delaying the
winner past the freeze; once the hold can release on first inventory,
the gate must stay closed until the queued checks have run.
* mini: gate the first-report release on single-master mode
releaseAllocOnFirstReport assumes the embedded volume server is the only
possible reporter of prior cluster state. A mini started with
-master.peers joins other masters, whose own volume servers can still be
on their way when the first report lands, so keep the regular
quiet-pulse hold there.
* master: check reported topology id on empty full reports
A HasNoVolumes heartbeat carries the same LastSeenTopologyId a populated
list does, but the mismatch freeze only ran for non-empty reports. An
EC-only veteran could therefore release the post-mint allocation hold
through NoteInventoryApplied while its stale topology id never reached
CheckReportedTopologyId.
* master: drain the topology-id stash inside the init-gate lock
CheckReportedTopologyId only held reportedTopologyIdsLock while stashing,
so a report in flight during SetTopologyInitPending(false) could land its
freeze after topologyInitPending cleared, letting a NextVolumeId slip
through the gap. The whole check now runs under the lock, and the gate is
released inside the same critical section as the drain.
* mini, master: pin and enforce the single volume-server reporter
isSingleMasterMode only checks master peers; SendHeartbeat still accepted
any volume server, so a late foreign report could carry prior cluster
state after the first-report release dropped the hold. Mini now pins the
embedded volume server's address and the master refuses registration from
any other reporter while ReleaseAllocOnFirstReport is set.
* master: admit the volume-server reporter before applying heartbeat state
* test: cover SendHeartbeat admission ordering
A refused reporter must not freeze growth or raise the id bound, and the
pinned reporter's stale topology id still must.
* mini: re-derive the single-reporter release on every run
The package-level master options survive in-process reruns, so a
multi-master mini following a single-master one must not inherit the
first-report release or a stale reporter pin.
* master: admit addressless first heartbeats under the reporter pin
An unregistered heartbeat with no Ip skipped the single-reporter
admission check but still reached the seen-bound raise and the
topology-id freeze before the delta-heartbeat skip. Admission now runs
for every unregistered heartbeat; an empty address can never equal the
pin.
* util: add listenOnLocalhost parameter to NewIpAndLocalListeners
Nil keeps the current behavior of also binding localhost when the host
is a specific address; all callers pass nil for now.
* command: add -ip.bind.localhost flag to opt out of the extra localhost listener
Binding a command to a specific -ip.bind address also opened a second
listener on localhost for every port, with no way to turn it off. Each
command that calls NewIpAndLocalListeners (master, master.follower,
filer, s3, iam, volume, sftp, mq.broker, mq.agent) now accepts
-ip.bind.localhost=false to bind only the requested address. Embedded
services get prefixed flags (s3.ip.bind.localhost, sftp.ip.bind.localhost,
iam.ip.bind.localhost), and the command-level flag also disables them so
a single switch turns off every extra listener the command opens.
* mini: only require all-interface port availability while the loopback listener is on
With -ip.bind.localhost=false each service binds just -ip.bind, so a port
held on another address no longer collides; the all-interface precheck
would reject or relocate usable ports. WebDAV, the Admin UI, and the
Admin worker gRPC (which binds -admin.worker.ip, not -ip.bind) never open
a loopback companion listener, so their checks now use the address they
actually bind.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* mq.broker: bind the gRPC listener to -ip.bind instead of all interfaces
The listener used an empty host so -ip was ignored for binding and the
localhost companion was never created, leaving -ip.bind.localhost with
nothing to control. Give the broker the same -ip/-ip.bind pair the other
servers use; embedded mode binds to -ip.bind like its peers.
* server: add -sftp.ip.bind.localhost and -iam.ip.bind.localhost switches
weed filer already exposes the prefixed opt-outs for its embedded sftp
and iam; give weed server the same pair so each extra listener can be
disabled individually while -ip.bind.localhost=false still turns them
all off.
* mq.broker: keep the default all-interface bind unless localhost is off
An unset -ip.bind used to mean the standalone broker listened on every
interface; defaulting it to -ip silently cut off multi-homed clients. Only
fall back to -ip when -ip.bind.localhost=false, where a specific bind is
needed for the opt-out to take effect.
* mini: validate the Admin HTTP port on the address it will bind
The port check always used -ip.bind while startMiniAdminWithWorker falls
back to loopback when Admin has no credential, so a loopback-held port
passed validation and killed the admin at startup.
* mq.broker: keep the all-interface default bind when -ip.bind is unset
-ip only becomes the fallback once -ip.bind.localhost=false needs a
specific address for the listener.
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* mini: validate the Admin HTTP port on its effective bind IP
When the Admin listener falls back to loopback (no credential or mTLS
configured), the pre-flight port check now probes that same address
instead of -ip.bind.
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* util: keep NewIpAndLocalListeners source-compatible for 3-arg callers
* command: drop the -ip.bind.localhost flag; an explicit -ip.bind drops the extra listener
Instead of an opt-out flag, the loopback companion listener now simply
follows whether the bind address was chosen: it is created only while
-ip.bind is unset and the bind address falls back to -ip. Passing
-ip.bind=<addr> binds each port to exactly that address.
Each service options struct carries a listenOnLocalhost *bool that
embedders (weed server, weed mini, embedded s3/sftp/iam in weed filer)
set from the parent's -ip.bind state, captured before the bind flag is
repointed or filled in by the -ip fallback; standalone commands derive
it from their own flag before the fallback runs.
iam and mq.agent gain a real -ip.bind flag so they have a way to pin a
single address. mq.broker keeps its new -ip.bind flag but drops the
-ip.bind.localhost fallback hack: empty still listens on all interfaces.
* server: default embedded mq.agent brokers to the broker's bind address
With an explicit -ip.bind the embedded broker no longer listens on
localhost, so an unset -mq.agent.brokers defaulting to localhost:17777
could not reach it. Default to <serverBindIp>:<mq.broker.port> (the
address the embedded broker binds), falling back to localhost when the
bind address is a wildcard. An explicit -mq.agent.brokers still wins.
Also fix the NewIpAndLocalListeners skip message to say localhost.
* server: bind embedded mq.agent to serverBindIp
The embedded agent's ipBind was never set, so startQueueAgent fell back
to the advertised -ip address and an explicit -ip.bind did not pin the
agent listener.
---------
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat: send EC shards in parallel per destination disk
One stream per destination disk. max_shard_transfers unset means no global cap, 0 sends shards sequentially, and N is a global cap that still keeps one stream per disk. The value is persisted through TaskPolicy. The maintenance form is a checkbox plus Max transfers because unset and 0 differ. The key is intentionally omitted from GetConfigSpec.
* ec shard distribution: treat negative max_shard_transfers as sequential
The config layer (normalizedMaxShardTransfers) clamps negative values to 0,
which selects sequential distribution. shardTransferCap diverged: a negative
cap that reached ShardDistributionOptions directly would have selected
uncapped parallel mode, the most aggressive setting. Align the options layer
with the same normalization so invalid input degrades to the conservative
mode.
* ec: serialize shard transfers to a server when a target leaves disk choice to the receiver
disk_id 0 is auto-placement on the receiving volume server, so an
auto-placed stream can land on a disk another queued stream is pinned
to. Grouping by (destNode, disk_id) alone let both run at once; collapse
all of that server's transfers under the destNode key instead.
* worker: clamp max_shard_transfers to the int32 wire range
ReadIntConfig returned the 0 fallback for values above MaxInt32, turning
a large positive cap into sequential mode. Read int64 and clamp into
range instead.
* worker: drop unused shard-transfer form helper
The maintenance form renders and collects max_shard_transfers in
plugin.templ JavaScript; the Go copy had no callers.
* worker: bound max_shard_transfers conversion under an in-range guard
---------
Co-authored-by: Chepurnov Simon <195125934+CHEPURNOV-S-S@users.noreply.github.com>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
* filer, volume: ignore Range when If-Range no longer matches
A GET with Range and a stale If-Range validator got 206 with the requested
bytes of the current file. RFC 9110 13.1.5: send the whole content instead,
so a resumed download restarts rather than mixing two versions.
* volume(rs): drop Range when If-Range no longer matches
Mirror the Go fix for the Rust volume server: evaluate If-Range against the
response ETag (strong comparison) or Last-Modified once the needle's
validators are known, then drop the range on a stale validator so the whole
current representation is sent. A dropped range on the RangeFromSource
strategy falls back to a full stream of the data-file region; buffered and
chunk-manifest paths already take Option<&str>.
* volume(rs): fail If-Range when the header value is not readable
* volume(rs): accept all HTTP-date formats in If-Range like Go
* server: cover alternate HTTP-date formats in If-Range test
---------
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
* master: release the post-mint allocation hold after a quiet pulse
DeferAllocation armed a fixed warmup window whenever the leader minted a
fresh topology id, so the first write on a brand-new cluster blocked for
the whole window even though no prior state could exist. The window
still has to cover a rebuilt master's veteran servers, which report on
their first post-reconnect heartbeat.
Track the last heartbeat that could carry prior cluster state — a server
joining, held volumes or shards, a nonzero id bound, or a foreign
topology id — and lift the hold one pulse after it goes quiet, capped by
the original window.
* master: make the post-mint allocation hold configurable
master.volume_growth.allocation_deferral overrides the warmup window a
leader holds volume id allocation for after minting a fresh topology
id. Unset keeps the existing window; "0s" disables the hold for setups
that know the cluster is fresh (test harnesses, single-node labs); any
other duration replaces the cap while the quiet-pulse early release
still applies.
* master: re-check the allocation deadline while it is held
NextVolumeId slept for the whole computed delay, so a heartbeat landing
mid-sleep that pushed the quiet deadline out was ignored — the pending
allocation could still run ahead of a straggler's report. Wait in short
slices, re-evaluating the deadline and the growth freeze each round.
* master: count only ids above the issued bound as prior-state evidence
Heartbeat echoes of volumes and shards this master allocated during the
hold refreshed the quiet window, so each growth allocation re-extended
the deferral until the hard cap. Evidence now requires an id beyond the
bound this leader has issued, a higher reported bound, or a foreign
topology id; a joining server (dn == nil) still counts once.
* master: reject an unparseable allocation_deferral instead of disabling the hold
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* master: do not count a skipped heartbeat as a joining server
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test: assert the allocation hold extends past the original quiet deadline
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* master: release the post-mint hold after two quiet pulses
One pulse proved short for a veteran that reconnects on a staggered
restart: with only five seconds of quiet, allocation could resume and
reuse ids before a slower server landed its first heartbeat. Two pulses
still halves the fresh-cluster first-write stall versus the full warmup
cap, and known-fresh setups keep the "0s" override.
* master: pick once more when assign times out as growth concludes
The pick loop's final attempt can predate the growth it triggered
concluding: growth registers its volumes in the layout before clearing
the pending flag, so a timeout landing on that boundary used to shed a
hard 406 over volumes that had just arrived. Pick once more after the
loop in both the http and grpc assign paths before deciding how to
shed.
---------
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* s3tables: scope the resolved anonymous identity instead of defaulting to allow
Catalog authorization fell back to the open default whenever the request
resolved to the configured anonymous identity, ignoring its grant. The
fallback now applies only when no principal was resolved (zero-config),
so anonymous requests are checked against the identity's configured
actions like any other principal. The S3 Tables API and the Iceberg and
Lance catalogs share this handler, so all three are covered.
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test: authenticate the catalog clients in the s3tables suites
Anonymous calls into the Lance and Iceberg catalogs are now scoped by
the anonymous identity's grant, so the suites that used to rely on the
anonymous pass authenticate like their real deployments do: x-api-key
for the Lance probes and REST drivers, client-credentials OAuth for
pyiceberg.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* worker(lance): authenticate namespace calls with the storage credential
Anonymous catalog calls are now scoped by the anonymous grant, so a
worker that holds an S3 credential sends it as x-api-key on the
namespace requests instead of relying on the anonymous pass.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test: authenticate the pushdown catalog connection in catalog_lancedb
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* lance: follow catalog redirects only within the same host
reqwest forwards custom headers on a follow, so a redirect to another
host would carry the x-api-key credential to a server it was never
meant for. Keep same-host hops for proxies that normalize the path or
upgrade the scheme.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* lance: warn when the catalog api key travels in cleartext
The x-api-key pair is a real storage credential: over plain http to a
non-loopback host the network can read it. Warn at client build so the
operator sees it; https and loopback stay quiet, and the key still
sends since an http catalog is a valid deployment.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* lance: require the same origin on catalog redirects
A same-host hop can still downgrade https to http and put the api key on
the wire; matching scheme, host, and port keeps the credential on the
listener it was issued for.
* lance: bound catalog redirect hops so loops fail fast
The custom redirect policy drops reqwest's default 10-hop limit, so a
catalog that redirects back to itself would hold each call open until
the request timeout. Keep the bound alongside the origin check.
* lance: gofmt the redirect helpers
---------
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* admin: keep read-only sessions out of the filer /etc tree
The file browser read routes are open to any authenticated session, so a
read-only login could download or view arbitrary filer paths. The filer
stores the credential and cluster configuration under /etc (identity JSON
with plaintext S3 secret keys, SSE-S3 keys, remote storage config), which
let a view-only session recover live secrets that the user API redacts.
Gate the read paths on the session role: read-only sessions get 403 for
/api/files/download, /view, /properties, /metadata on /etc paths, the
browser page redirects away, and recursive metadata exports skip the /etc
subtree for read-only sessions.
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* admin: only reject real .. path components in file browser paths
path.Clean already resolves interior parent references, so any '..'
surviving in the cleaned path can only be a leading escape from a
relative input or part of a valid name. Matching on the substring
rejected legitimate entries like /buckets/reports/report..csv.
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* admin: cover the read-only /etc export exclusion with a test
Pull the excluded-prefix selection into exportExcludedPrefixes so the
role-dependent /etc skip is exercised directly — the cleanReadPath
tests alone would still pass if the exclusion were dropped.
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* s3api: stop echoing stored tags and internal attributes on GET/HEAD
setResponseHeaders copied every entry.Extended key into the response,
only filtering xattr- and x-seaweedfs-. Stored object tags
(X-Amz-Tagging-*) and the gateway's own attributes (Seaweed-X-Amz-Acl,
-Owner, -Version-Id, -Object-Lock-Mode, ...) therefore leaked to any
caller holding s3:GetObject, and to anonymous readers on public-read
objects, without s3:GetObjectTagging or READ_ACP.
Tag count, user metadata, storage class and the real S3 headers
(x-amz-version-id, x-amz-object-lock-*) are unaffected.
* s3api: match stored tag and internal attribute keys case-insensitively
GitHub is retiring ubuntu-22.04; move all 94 runs-on pins across 40 workflows to ubuntu-24.04. rust_binaries_release: the arm64 cross step was jammy-era — noble keeps apt sources in deb822 ubuntu.sources (the sources.list edit was a silent no-op that left amd64 repos open to arm64 and 404'd) and ports suites must say noble. Pin deb822 stanzas to amd64 and add noble ports entries.
* test: dump trino container state and logs when it exits before ready
When the Trino container dies during startup, waitForTrino only
reported the docker exec error ("container is not running"), which
gives no hint whether the JVM was OOM-killed, the image broke, or the
config is bad. Capture docker inspect exit state and the container
logs in the failure message so runner-side flakes are diagnosable
from the test output alone.
* test: cap trino container memory at 4g in catalog tests
Trino's jvm.config sizes the heap at 80% of host RAM. On a loaded
self-hosted runner the JVM attempts to claim most of system memory
at boot and the container is killed ~15s after docker run — the
flake seen in CI today. Bounding the container to 4g gives the JVM
a fixed ~3.2g ceiling (observed usage is ~750mb), immune to host
memory pressure.
* test: keep docker errors in trino container diagnostics
Report docker inspect/logs failures instead of dropping them, and
include the last docker exec output in the container-dead fatal so a
missing container still explains what happened.
* test: detect trino container exit in readiness fallback
When the polling loop times out while the container still appears
alive, the trino --version fallback may itself fail because the
container has since exited. Re-check its output for the
exited-container signature and route through the diagnostic dump.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test: dump trino container diagnostics on readiness timeout too
The alive-but-never-ready timeout path reported only the last docker
exec error, losing the container state and logs that the dead-container
path already captures.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test: prefixed directory listing with start below the prefix
A listing constrained to prefix "item-b" and resumed from "item" must
return item-b.versions. The seek-based stores return an empty page
because the iterator lands before the prefix range and breaks on the
first non-matching name.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* filer: clamp prefixed-listing seek to the prefix range
Seek-based stores (leveldb, leveldb2, leveldb3, rocksdb, tikv, etcd)
started the scan at startFileName and stopped at the first key outside
the prefix. A startFileName below the prefix landed the iterator on a
non-matching sibling first, so the listing returned an empty page even
though matching entries followed. etcd had the mirror problem: a range
beginning below the prefix end leaked non-matching entries. Begin the
seek at max(startFileName, prefix), which is what foundationdb and the
sql stores already do.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* filer(hbase): honor startFileName lower bound in prefixed listings
The scan starts at the prefix but only skipped the exact startFileName,
so entries sorting before it were returned again on resume.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* filer: keep paging a prefixed listing while matches can still follow
The fallback for stores without native prefix support stopped paging as
soon as the last scanned name reached the prefix, dropping prefix
matches that continue on the next page.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* filer: return the last delivered name from the prefixed-listing fallback
The fallback for stores without native prefix support pages raw entries
and filters client-side. Keep the scan cursor and the returned cursor
separate: resuming must continue after the last delivered match, not the
last scanned name, or matches still inside the current raw page are
skipped on the next page. Also keep paging while the scan cursor is
inside or below the prefix range instead of stopping at the first name
that reaches the prefix.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* filer(hbase): start prefixed-listing scans at the resume key
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* volume: overlap the local needle write with the replica fan-out
ReplicatedWrite wrote the needle to local disk and then fanned it out to
the replicas serially, so a replicated write cost local + replicas. The
needle is already fully buffered, so the two can run at once. Snapshot
the fields the fan-out reads first (the local write stamps the volume
TTL and updates needle flags, so the replica goroutine must not touch
the shared needle), then run the local write and the replica fan-out
concurrently and ack only after both. Where local disk and replica
network are separate resources the write latency drops to max(local,
replicas) instead of their sum; co-located on one disk the two writes
contend, so this helps distributed placements, not single-host tests.
* volume: return the local isUnchanged from the write goroutine
* volume: trim overlap-write comments
* volume-rs: overlap the replica fan-out with the local needle write
* volume: test the overlapped local write and replica fan-out
* volume: decode replica pairs only when replicas exist
A single-copy or incoming replicate write never uses the decoded pair map, so a malformed Pairs blob only burned an error counter and a log line. Gate the decode on replicaCount.
* volume: release the stalled replica on test failure paths
An early t.Fatal left the httptest handler blocked on releaseReplica, so the deferred Close hung instead of reporting the failure. Release idempotently ahead of the server close.
* test: mismatched If-None-Match serves the object over If-Modified-Since
The precedence case asserted the old If-Modified-Since-first ordering and
expected 304. The volume servers now evaluate If-None-Match first, so a
non-matching tag must return 200 with the body even when If-Modified-Since
would allow 304 (RFC 9110 13.1.3).
* s3: authorize POST policy uploads through the shared object check
doesPolicySignatureMatch gated a verified POST Object signature on the
identity's legacy Write action, which cannot represent attached IAM
policies, STS session policies, or key-scoped grants. Users authorized
only by an attached policy (e.g. s3:PutObject on bucket/*) were denied
with AccessDenied even though the equivalent PUT succeeds.
The handler already runs AuthorizeObjectWrite against the real object
key after the form's x-amz-* fields are promoted to headers, so the
coarse check only shadowed the full evaluation. Signature verification
now just returns the identity; authorization stays in one place.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test: POST Object authorizes attached-policy identities, still denies others
Cover the signature check and AuthorizeObjectWrite for an identity whose
only write grant comes from an attached IAM policy, plus the deny leg
for an identity whose attached policy omits s3:PutObject.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
The precedence case asserted the old If-Modified-Since-first ordering and
expected 304. The volume servers now evaluate If-None-Match first, so a
non-matching tag must return 200 with the body even when If-Modified-Since
would allow 304 (RFC 9110 13.1.3).
ListDirectoryEntries fetched every member with its own GET, so a page of
1024 entries cost 1024 round trips. Fetch the values in pipelined batches
of 128 as the loop reaches them and walk the replies in order. Each reply
keeps its own error, so missing and expired members are cleaned up and
skipped exactly as before, and an unreadable member still ends the
listing at that name. Batching keeps a caller that lists with a huge
limit and stops after a few entries from reading the whole directory.
Co-authored-by: Patrick Schmidt <1310072+Woellchen@users.noreply.github.com>
Co-authored-by: user.email <chrislusf@users.noreply.github.com>
weed filer lost -redirectOnRead in 9b3109a5d (1.59), which also removed
the chart value, but 2bed15f1d added it back. Setting it makes the filer
exit with "flag provided but not defined".
* shell: run volume.configure.replication in parallel per volume server
Volume servers were configured one after another, and each server's
result overwrote the previous error, so a failure on one server was
lost when a later server succeeded. Run one job per volume server,
bounded by -maxParallelization like the other shell commands, and
return the errors from all servers.
VolumeConfigure unmounts and remounts the volume, so replicas of the
same volume on different servers take turns instead of all going
offline at once.
Co-authored-by: NyaMisty <5344431+NyaMisty@users.noreply.github.com>
* shell: test that volume.configure.replication overlaps servers up to the limit
* shell: assert every server ran in the parallel configure-replication test
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: NyaMisty <5344431+NyaMisty@users.noreply.github.com>
Co-authored-by: chrislusf <chrislusf@users.noreply.github.com>
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
The S3 Remote Cache tests exercise the remote-storage S3 client, but the
path filter did not include weed/remote_storage, so a tagging change
there can regress the suite without running it on the PR.
Generated with [Devin](https://devin.ai)
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* remote_storage/s3: send only real object tags to the remote
WriteFile and toTagging copied every entry.Extended attribute into the
remote object's tag set, including Seaweed-internal keys such as
Seaweed-X-Amz-Acl whose values are not legal S3 tag values. Remotes
that validate x-amz-tagging now reject these uploads with InvalidTag.
Only X-Amz-Tagging-* attributes are S3 tags: filter to that prefix,
strip it, and encode through s3TaggingHeader so keys and values are
properly escaped.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* remote_storage/s3: skip remote tagging call when the tag set is unchanged
UpdateFileMetadata fired PutObjectTagging on any Extended diff, but most
attribute changes are not tag changes. Compare the effective tag sets and
return early when they match.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* remote_storage/s3: assert WriteFile errors in tag-header tests
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* filer.backup: retry a failed checkpoint read instead of replaying from the epoch
Fixes#12035
* test: reset sink.local.directory in backup offset test cleanup
The new test sets sink.local.directory on the shared viper but only
reset the enabled flag, leaking a key that points at a deleted
t.TempDir() to later tests in the package.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test: cover the retried attempt re-reading the checkpoint
The fake filer now fails only its first KvGet, and the test runs a
second doFilerBackup attempt asserting the checkpoint is re-read and
the attempt proceeds past the offset stage. LookupDirectoryEntry is
stubbed so the SSE-S3 KEK probe returns a miss instead of retrying
10x2s.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: user.email <chrislusf@users.noreply.github.com>
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* s3: check pattern '*' before literal equality in matchWildcardString
When the target string carried a '*' at the offset of the pattern's '*',
the literal-equality case consumed it as a one-character match and no
backtrack point was recorded, so the comparison returned false. A Deny
Resource carrying a policy variable, or an Allow carrying NotResource,
could then be sidestepped by an object key containing '*' at the aligned
offset.
* test: cover literal asterisks in wildcard match targets
---------
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Same startup-timing flake class as the Makefile waits: volume/filer
readiness at 30s is marginal on loaded runners — observed failing on a
github-hosted runner too (TestPOSIXCompliance +
TestExternalPOSIXSuites). The process-death early return still reports
real startup failures quickly.
* s3: keep ListObjectsV2 continuation tokens round-tripping XML-unsafe keys
A page ending on a key XML 1.0 cannot carry (for example one containing
\x01) produced a NextContinuationToken from that raw key. The XML
encoder replaced the control characters with U+FFFD, so a client resuming
at the token restarted after the mangled name and skipped every key
sorting between it and the real one. encoding-type=url encoded the keys
but not the token, so it hit the same skip.
Encode the token as base64url behind a prefix when the key is not valid
XML text; keys that survive XML unchanged keep the bare key as the token,
so their tokens are byte-identical to before. The handler decodes the
prefix back to the raw key before listing.
Fixes#11990
* s3: keep undecodable base64: list tokens resuming at the raw key
* s3: round-trip ListObjectsV1 markers over XML-unsafe keys
* s3: round-trip ListObjectVersions key markers over XML-unsafe keys
* s3: round-trip ListMultipartUploads key markers over XML-unsafe keys
* s3: decode only tokens whose key could not have stayed bare
* s3: document and pin the encoded-wins rule for ambiguous list tokens
A token like base64:AQ reads as both a legacy bare key and the encoded form of an XML-unsafe key. The encoded reading already won in code; state why and cover it so the choice is not reverted as a false simplification.
* s3: keep request-marker echoes in the response's wire encoding
The decoded marker/key-marker is a raw key; echoing it writes XML-unsafe
bytes straight into the response, where the encoder turns them into
U+FFFD. Re-encode each echo through encodeListContinuationToken so the
client reads back the token it sent, and apply urlPathEscape for V1 when
encoding-type=url is active, matching the listed keys.
* s3: wire-encode the V1 NextMarker echo like the request marker
* s3: pin resuming an XML-unsafe key past a sibling directory
* s3: wire-encode ListMultipartUploads markers and keys under encoding-type=url
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* s3: unescape url-encoded marker echoes on list resume
Under encoding-type=url the V1 and ListMultipartUploads responses emit
marker echoes through urlPathEscape, and clients pass them back verbatim.
The incoming marker was used as the resume position without undoing that
escaping, so resuming after a url-unsafe-but-XML-safe key like "a b"
(marker "a%20b") restarted the listing at the literal escaped text.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* s3: assert the full decoded V1 sequence in the marker round-trip test
The test only checked the page count, so a regression that skipped one
key while repeating another still passed. Requesting encoding-type=url
keeps the XML-unsafe keys legible for the decoded comparison and
exercises resuming with the escaped marker echo.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* s3: only unescape markers in the canonical urlPathEscape form
A url-encoded request marker like a%41 cannot be encoder output, since
urlPathEscape never emits %41 for 'A'. Unescaping it anyway resumed at
aA and skipped every key sorting between a%41 and aA, including a%42.
Require the unescape to re-escape to the received marker so only
verbatim echoes decode; anything else stays the raw key the client sent.
* s3: pin a raw-key marker that looks like an escape sequence
* s3: cover the marker-echo decode guard with unit cases
* s3: wire-encode the ListObjectsV2 StartAfter echo under encoding-type=url
AWS encodes Delimiter, Prefix, Key, and StartAfter in V2 responses when
url encoding is requested, but StartAfter was echoed raw. The echo now
takes urlPathEscape and the incoming start-after param runs through
decodeMarkerEcho, so a client resuming from the emitted form lands on
the raw key.
* Merge master and reconcile ListMultipartUploads marker encoding
Resolve the overlap with the merged encoding-type=url pass: keys and
markers now go through the single post-pass so nothing is url-escaped
twice. Markers are still token-encoded when the response is not
url-encoded, and the key-marker decode checks the query-escaped wire
form the endpoint actually emits.
* s3: echo the decoded start-after under encoding-type=url
The escaped echo of the raw input is not a fixpoint: a client resubmitting
the echoed StartAfter decoded it to the literal escape text and resumed at
the wrong position.
* s3: keep url-mode multipart key-marker decodes off the token path
Under encoding-type=url the emitted KeyMarker/NextKeyMarker are the
url-escaped raw key, never a base64-prefixed token, so the request decode
must unescape only. Running the token decode afterward misread a literal
"base64:..." key as an encoded token and resumed at the decoded bytes,
skipping the keys in between.
---------
Co-authored-by: Chris Lu <chris.lu@gmail.com>
Co-authored-by: user.email <chrislusf@users.noreply.github.com>
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* mq/kafka: wait for a filer to register instead of panicking on an empty list
NewSeaweedMQBrokerHandler queried the masters for filers once, ~2 s after
start, and handed an empty list to NewFilerClientAccessor, which panics. A
fresh filer registers only after its empty-membership confirm window
(10 s), so a gateway started with a fresh cluster crashed (Kafka Quick
Test, kafka-gateway exit 2). Retry while the list is empty for up to 60 s,
then return an error.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
* mq/kafka: wait out a transient-empty broker listing too
A fresh SMQ broker registers with the masters on the same announce
delay as a filer, so the one-shot broker lookup could fail startup the
same way. Generalize waitForFilers into waitForClusterNodes and bound
both discoveries by the same window; the empty-broker check becomes
unreachable and drops away.
* mq/kafka: wait out the broker health check like node discovery
A broker announces to the masters before it has acquired its balancer
lock, so the gateway's three-attempt probe could still fail fast on a
fresh cluster. Retry FindBrokerLeader for the same window the caller
waits out empty listings.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
Co-authored-by: user.email <chrislusf@users.noreply.github.com>
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* ci(s3tables): tolerate stale FUSE mounts in disk diagnostics
* ci(s3tables): bound df diagnostics with timeout, explain once
A stale FUSE mount can also hang statfs, not just fail it, so wrap df in
timeout 15; one comment at the first site covers all sixteen.
* ci: tolerate stale FUSE mounts in df diagnostics elsewhere
The same self-hosted runners serve the other workflow files; a bare
df -h there can stall or fail a job the same way.
* ci: unmount disconnected FUSE test mounts before jobs start
* ci: skip FUSE cleanup when the variant checkout is skipped
* ci: reclaim the crashed run's test dir once its stale FUSE mount is detached
* ci: stop tracking python bytecode for the FUSE cleanup action
The helper's __pycache__ slipped into the tree; ignore it so local
test runs cannot re-add it.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* ci: run the FUSE cleanup action's unit tests on action changes
test_cleanup.py was never wired into CI, so a regression in cleanup.py
would ship silently. A path-filtered workflow runs it on every change
under .github/actions/.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* ci: match every FUSE test mount layout in the cleanup action
The candidates list only matched seaweedfs_fuse_test_*/mount, so stale
mounts left by fuse_failover and fuse_p2p runs (mountN leaves under
seaweedfs_fuse_* dirs) and the posix-compliance workflow's fixed
/tmp/seaweedfs/mount were never reclaimed and could still wedge df.
* ci: run the FUSE mount cleanup in the fuse_failover and fuse_p2p workflows
---------
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat: Free Volume Deadband (M) for EC Place nodes and racks
|ΔfreeSlots| ≤ M is a tie; remainder is liveShards. M=0 stays strict
freeSlots. M is free volume slots (× dataShards). pickNodeInRackEligible
and pickRackEligible share the rule. Worker plugin config
free_slots_deadband, default 0. Not on the write path.
* fix: anchor EC freeSlots deadband on class maxFree
Pairwise |Δfree|≤M vs the current winner is not transitive: 120→110→100
at M=10 slots picked 100. After evenness, rank by maxFree, keep
|free-maxFree|≤M, then min liveShards. Shared by node and rack. M=0 is
still only maxFree then live.
* worker: add free_slots_deadband to ErasureCodingTaskConfig
* erasure_coding: persist free_slots_deadband through TaskPolicy
ApplyMaintenanceConfigFromToml accepted the key but the admin save path
round-trips config through ToTaskPolicy, so the value was dropped on
reload; serialize it like the other EC settings.
* erasure_coding: bound free_slots_deadband at its declared maximum
The TOML loader and the plugin worker config only clamped negatives, so
a value above the spec's 100000 could reach the int32 TaskPolicy field
and wrap; cap at 100000 on both entry paths and declare MaxValue on the
plugin field.
* erasure_coding: document free_slots_deadband in scaffold admin.toml
---------
Co-authored-by: Chepurnov Simon <195125934+CHEPURNOV-S-S@users.noreply.github.com>
Co-authored-by: user.email <chrislusf@users.noreply.github.com>
* s3: accept a self-copy that changes the storage class or encryption
AWS changes an object's storage class or encryption by copying the
object onto itself with the new x-amz-storage-class or SSE headers, and
only rejects a self-copy that changes none of metadata, storage class,
website redirect location or encryption. CopyObject rejected every
self-copy without REPLACE metadata or tagging directives as
InvalidRequest, so these requests failed.
* s3: reject an invalid x-amz-storage-class on CopyObject
A self-copy with a bad storage class used to fail as InvalidRequest;
once such copies are admitted the value must be validated like PutObject
does, or it would be stored as opaque metadata.
* test: cover the invalid storage class rejection on self-copy
* s3: only a storage class change admits a self-copy
Admitting encryption changes on a self-copy is unsafe: the metadata-only
path compares only KMS key IDs, so a new encryption context or bucket-key
setting is advertised over the old data key, and SSE-KMS key rotation
onto the same entry keeps the old data key.
Count only a requested storage class that differs from the object's
(STANDARD when none is stored) as a change, so a self-copy to the
current class or with only encryption headers is rejected as before.
Such a self-copy updates the object in place and has to keep its
encryption; one that would re-encrypt it or uses SSE-KMS is rejected
too.
---------
Co-authored-by: Wleeaf <179125857+wleeaf@users.noreply.github.com>
Co-authored-by: user.email <chrislusf@users.noreply.github.com>
* topology: drop a duplicate-vid report whose own copy is already gone
flagDuplicateVolumeId runs on a goroutine after registration, so by the
time it executes the reporter's copy may have been unregistered (a
finished move drops the losing copy). The surviving copies are then
legitimate, yet the stale report still quarantined them and stored a
phantom (node, vid) key for the departed reporter. Skip the marks when
the reporter no longer serves the volume under its own layout key.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(topology): cover a stale duplicate report against a lone survivor
The existing tests only flag the conflict before the losing copy is
unregistered. Run the delayed duplicate check after the reporter's copy
is already gone and assert nothing is quarantined and the survivor stays
writable, so removing the stale-report guard fails the test outright
instead of needing a lucky schedule.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(topology): cover the enforceQuarantine-driven release retry
A failed writable push re-arms the record, and the next volume report
drains that retry through enforceQuarantine itself - a different entry
point than the pulse-driven pass the other tests exercise.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* filer: match If-Match and If-None-Match against "*" and ETag lists
checkPreconditions compared the whole header as one ETag, so "*", a
list of ETags and a weak W/ tag in If-None-Match never matched
(RFC 9110 13.1.1, 13.1.2).
* volume: check If-None-Match before If-Modified-Since and match ETag lists
A volume GET answered 304 from If-Modified-Since even when If-None-Match
named a different ETag, and compared If-None-Match as one exact string,
so "*", lists and weak tags never matched (RFC 9110 13.1.2, 13.2.2).
* filer: split etag lists only outside quoted tags
A comma inside a quoted opaque tag is a legal etagc, not a list
separator; strings.Split cut such tags into fragments that
CanonicalizeETag could turn into a false match.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* filer: match repeated If-Match and If-None-Match header lines
Header.Get reads only the first field line, so a match on a later
repeated header was missed; join all field values like any other
comma-separated list.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* volume: match repeated If-None-Match header lines
Header.Get reads only the first field line; join all field values so a
match on a later repeated header is seen. Also covers a comma inside a
quoted opaque tag.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* volume: send the etag on a 304 like the filer does
RFC 9110 15.4.5 lists ETag among the fields a 304 must carry when a 200
would send it; SetEtag now runs before the notModified check.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* volume: mirror the conditional-GET fixes in the rust server
not_modified_response now checks If-None-Match before If-Modified-Since,
matches "*", weak tags, quoted commas and repeated field lines through
the same list rules, and sends the etag on a 304.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* volume: treat an empty If-None-Match like the Go server does
* filer: ignore empty elements in etag precondition lists
* volume: ignore empty elements in etag lists like the filer
* filer: let a present-but-empty precondition line suppress the date check
* volume: gate If-Modified-Since on a present If-None-Match line
* volume: gate If-Modified-Since on header presence like the Go server
---------
Co-authored-by: user.email <chrislusf@users.noreply.github.com>
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: chrislusf <chris.lu@gmail.com>
* master: read the writable-volumes cap from master.toml
-maxWritableVolumes becomes the master.max_writable_volumes config key,
so the new master limit is set the same way as other master options
instead of a flag wired into every command that embeds a master.
* master: name the writable cap master.max_writable_volumes_per_layout
The cap applies per volume layout (collection + replica placement + ttl +
disk type), so the key names its scope instead of reading as a global cap.
* master: keep an embedder's MaxWritableVolumes when the key is unset
NewMasterServer overwrote option.MaxWritableVolumes with the viper
value, which reads as 0 whenever master.toml omits the key, silently
dropping a cap a Go caller supplied through MasterOption. Seed the
default from the option instead so the field survives an absent key;
an explicit config entry still wins.
* master: drop stray nested-checkout gitlinks
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* filer: match If-Match and If-None-Match against "*" and ETag lists
checkPreconditions compared the whole header as one ETag, so "*", a
list of ETags and a weak W/ tag in If-None-Match never matched
(RFC 9110 13.1.1, 13.1.2).
* filer: split etag lists only outside quoted tags
A comma inside a quoted opaque tag is a legal etagc, not a list
separator; strings.Split cut such tags into fragments that
CanonicalizeETag could turn into a false match.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* filer: match repeated If-Match and If-None-Match header lines
Header.Get reads only the first field line, so a match on a later
repeated header was missed; join all field values like any other
comma-separated list.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* filer: ignore empty elements in etag precondition lists
* filer: let a present-but-empty precondition line suppress the date check
---------
Co-authored-by: user.email <chrislusf@users.noreply.github.com>
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: chrislusf <chris.lu@gmail.com>
* storage: hold a below-minimum fsync batch open for stragglers
When the durable-write queue drained, the worker flushed whatever it
held, so low-concurrency fsync writes collapsed to one sync per
request -- a seek plus a rotation on rotational media no matter the
payload. Once the queue empties with the batch still below the
configured minimum bytes and requests, the batch now stays open up to
SyncBatchOption.Wait, armed once per batch so the added latency stays
bounded. DiskLocation carries the option and defaults it; Store exposes
a setter the server calls once at startup.
* volume server: expose the fsync batch window as -fsync* flags
-fsyncMinBatchBytes, -fsyncMinBatchRequests and -fsyncBatchWait carry
the operator's batching floor into every disk's SyncBatchOption.
-fsyncBatchWait=0 restores the previous flush-on-drain behavior.
* server/mini: register the fsync batch window flags
weed server and weed mini build their own VolumeServerOptions and left
the three new fsync flag pointers nil, so startVolumeServer panicked
dereferencing them at startup.
* storage: bound the fsync batch window by Wait while the queue refills
Once the straggler timer was armed, a steady trickle of queued writes
kept taking the len(requests)>0 continue path, so the first write could
wait past Wait until the size or count cap flushed the batch. Track the
armed deadline and flush once it passes even with requests buffered.
* storage: sync the straggler test on the worker taking the first write
The fixed sleep let both writes queue together (passing under the old
flush-on-drain behavior) or the straggler arrive after the window closed
(flaky failure). Poll the drained queue instead so the second write can
only land inside the armed window or a later batch.
* volume server: default the fsync batch window when its options are unset
weed server and weed mini build VolumeServerOptions without the fsync
flag pointers, and outside embedders can too. Resolve the batch window
from the defaults so startVolumeServer never dereferences nil options.
* storage: drive the batch-window test through an injectable timer
The window is armed by the worker now, so the straggler provably joins
an open batch and expiry is fired by hand instead of racing a sleep.
* seaweed-volume: accept -fsyncMinBatch/-fsyncBatchWait flags
Same flags and defaults as weed volume so the Rust server's durable
write batching can be tuned identically.
* seaweed-volume: hold a below-minimum fsync batch open for stragglers
Mirrors the Go worker's batchWait: once the queue drains under the
minimum batch, the window stays open for stragglers and an expired
window still bounds the batch while the queue refills. Paused-time
coverage replaces any sleep-based assertion.
* seaweed-volume: sync the straggler test on the armed batch window
The test fired both writes together, so the straggler could already be
queued when the worker drained — a regression that drops mid-window
writes from the open batch would still pass. Wait until the worker
arms the window before sending the last write (Go's test polls the
drained channel for the same guarantee).
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* seaweed-volume: reject fsyncMinBatchBytes that does not fit usize
---------
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* s3: allow up to 50 tags in a bucket tag set
PutBucketTagging validated the tag set with the object tag rules, which
allow 10 tags. AWS allows a bucket up to 50 tags, so tag sets of 11 to 50
tags were rejected with InvalidTag. Validate bucket tags against 50 and
keep the limit of 10 for objects and S3 Tables.
A tag set that repeats a key is rejected with InvalidTag, as AWS does,
instead of keeping one of the values, so 51 tags with a repeated key no
longer pass as 50.
* s3: reject duplicate tag keys in PutObjectTagging
The tag set collapses to a map, so a repeated key was silently kept once.\nMatch the bucket tagging check and answer InvalidTag instead.
* s3: reject empty tag keys
AWS requires a tag key to be at least one character; the shared\nvalidator only checked the 128-char upper bound.
* s3: reject duplicate tag keys in the tagging header parser
parseTagsHeader (the x-amz-tagging path used by CopyObject with
TaggingDirective=REPLACE) silently kept the last value when a key
repeated, while the PutObject and multipart header paths and the new
XML checks return InvalidTag for the same input. A repeated key after
URL decoding now fails the same way.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* s3: validate x-amz-tagging tags on PutObject before storing the entry
The header path only rejected duplicate keys, so an upload could store
more than ten tags or keys and values that fail S3 tag rules. Run the
parsed set through ValidateTags like the XML tagging handlers do.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* s3: validate x-amz-tagging tags on multipart initiate
ParseS3Metadata stored header tags with no tag-set validation, the same
gap putToFiler had. Multipart uploads now reject oversize or malformed
tag sets with InvalidTag.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* s3: count tag length in characters, not bytes
* s3: validate object tags before uploading chunks
* s3: reuse parseTagsHeader and drop raw header from warnings
---------
Co-authored-by: Wleeaf <179125857+wleeaf@users.noreply.github.com>
Co-authored-by: user.email <chrislusf@users.noreply.github.com>
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
refreshUrls invalidates every cached location of a volume after a failed
read and re-looks it up through the reader's lookup function. With the
cache-only masterClient.LookupFileId, one failed read to one replica left
the volume missing from the filer's cache until restart, so metadata-log
replay skipped it. Use GetLookupFileIdFunction like the other read paths.
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
* iam/oidc: wrap the jti replay cache in its own type
The replay-protection state lived as ten fields and nine methods spread
across OIDCProvider. jtiReplayCache now owns the store, count, bound,
sweep scheduling, and cleanup worker; the provider keeps a single jti
field and delegates configure, consume, stop, adopt, and merge.
* iam/oidc: lazily allocate the jti replay cache
A zero-value OIDCProvider left jti nil, so Initialize, single-use
validation, Shutdown, and the replay-state handoffs panicked on a nil
cache. Allocate the cache through a sync.Once-guarded getter instead.
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* topology: serialize quarantine records with their layout marks
A quarantine publishes a (node, vid) record and marks the replica read-only at the layout level; release and heartbeat flag updates must observe that pair as one state. Store-then-mark let a concurrent release delete the record before the mark landed, stranding the surviving replica read-only; mark-then-store let a heartbeat clear the mark before the record was visible, returning a conflicting replica to the writable list.
quarantineLock now covers every record<->mark pair: flagDuplicateVolumeId's quarantine, releaseResolvedQuarantine's undo, the isQuarantined-gated marks on registration, and the quarantine-folded read-only flag writes on heartbeat paths. Remote server pushes stay outside the lock; a push that lands after the record was released is undone, and a failed release push re-adds the record for a later retry.
The release test now waits for in-flight async flags to settle instead of asserting on a single read.
* topology: fold quarantine into RegisterVolume's flag rebuild
RegisterVolume rewrites every replica's location read-only flag from what
its node last reported, and reported flags deliberately exclude quarantine.
Any registration of a shared vid -- a reconnected replica, a new copy, the
self-heal and delta-heartbeat re-register paths -- therefore cleared the
quarantine mark on every replica in the location, while the gate afterwards
only re-marked the registering node. Two call sites had no gate at all.
RegisterVolume now takes an isQuarantined predicate and evaluates it under
accessLock for every replica it touches, so the mark is never dropped in
the first place: a release unmarks after the rebuild, and a quarantine
mark re-applies on the next rebuild, with no window between.
* topology: keep quarantine when a vacuum commit re-adds a replica
SetVolumeAvailable re-adds the replica into the location list; a re-added
slot starts with a cleared read-only flag, so a quarantined replica came
back flag-free and the next ensureCorrectWritables could return a
conflicting volume to the writable list. Pass the quarantine state in and
set the flag back under the same accessLock, with the caller holding
quarantineLock so release cannot interleave between the check and the mark.
* topology: merge quarantine records instead of overwriting them
enforceQuarantine wrote back the state copy it loaded before its remote
push, so a concurrent writer's enforced=true was lost: the server stayed
read-only while the record said unenforced, and a later release skipped
the writable push it owed. The write-back now reloads the record under
quarantineLock and merges -- enforced wins from either side, reported is
taken from the newer pre-enforcement observation only while no push has
landed yet.
All three compensating writable pushes had the same hole in reverse: a
failed un-quarantine left the server read-only with no record to retry
from, and release's restore blindly overwrote any record a fresh flag
pass had created in between. They now go through restoreQuarantineRecord,
which adds the record only when absent and otherwise sets enforced on
the current one, preserving its reported flag.
* topology: hold quarantineLock over registration's quarantine check-and-mark
RegisterVolume reads the quarantine record and marks the replica
read-only under accessLock, but the pair ran without quarantineLock.
A release could push writable, watch the record get re-read, and then
delete it: the replica stayed read-only with no record left to clear,
and unchanged writable heartbeats skip SetReplicaReadOnlyFlag so the
mark never repaired. Holding quarantineLock around the check-and-mark
in all three production callers serializes it against the release
clear-and-delete.
* topology: serialize remote quarantine pushes behind a record recheck
releaseResolvedQuarantine deletes the record under quarantineLock and
sends the writable request after unlock. A conflict flagged while that
request is in flight creates a fresh record and marks the replica
read-only again, and the older writable request can still land last,
leaving the server writable under an active quarantine. The reverse
flap strands a stale read-only mark the same way.
Route every QuarantineVolumeFn call through pushQuarantine, which
holds quarantinePushLock across the remote call and re-checks the
record under quarantineLock right before sending: a read-only push
requires the pair to exist, a writable push requires it to be gone.
A skipped push means the decision already moved on, so the final
server state always follows the newest record.
* topology: serialize quarantine pushes per replica, not globally
A single quarantinePushLock held across the remote mark call let one slow
volume server (up to the 10s client timeout) delay pushes for unrelated
replicas. Keying the lock by (node, vid) keeps pushes to one replica
ordered in record-decision order while pushes to others run in parallel.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* topology: restore the layout read-only mark with a released record
A failed writable push re-armed the quarantine record but left the
replica marked writable in the layout, so PickForWrite could route to a
replica the server still holds read-only.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* topology: wait out async duplicate checks in the release-retry test
TestFailedReleasePushRestoresQuarantineMark asserts layout marks around a
release pass, but the two RegisterVolumeLayout calls each start a
background flagDuplicateVolumeId that can still be mid-flight. A late
mark could land after the final release and fail the writable assertion
even though the code behaved correctly. Track the async passes in a
WaitGroup and drain them before the explicit flag call so the rest of
the test is deterministic.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* topology: keep quarantine when a replica's layout key is unreadable
* topology: keep a failed writable push's restore ordered under the push lock
* topology: retry the quarantine release on every duplicate check
* ci: isolate Java and S3 test servers on persistent runners
* ci: drop the duplicated -ip flags in the lifecycle test server
The bind flags were already on the command; the isolation pass added a
second pair.
* topology: retry a failed quarantine writable push on the next report
* topology: retry pending quarantine releases off the leader pulse
A failed writable push re-arms the quarantine record with a retry that only
enforceQuarantine fires, and that runs only for volumes a heartbeat carries.
Once the server has reported the replica read-only it drops out of the delta
reports, and restoreQuarantineRecord keeps the stored flag matching the
server's so the digests agree and no full list is requested - the pending
release then never retries and the idle volume stays read-only forever. Run
the same release pass from the leader's pulse loop so a pending retry no
longer waits on a report.
* topology: exercise the pulse-driven quarantine release retry
---------
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* s3: evaluate bucket CORS for path-style OPTIONS on the bucket root
A path-style preflight for /bucket (no trailing slash) missed the bucket
OPTIONS route, whose template is /{bucket}/, and was answered by the global
OPTIONS handler from the global AllowedOrigins instead of the bucket CORS
configuration. Register the bucket OPTIONS route without a path matcher, and
keep OPTIONS for the health probe paths on the service-level handler ahead of
each path-style router.
Fixes#12025
* test: drop the issue reference from the bucket-root preflight test comment
---------
Co-authored-by: Wleeaf <179125857+wleeaf@users.noreply.github.com>
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* master: never auto-admit raft peers on a single-node configuration
A master started without cluster peers (peers=none, or no peers so the
list holds only itself) declares a one-node cluster, yet any foreign
master that registered over KeepConnected was admitted as a second
voter. The resulting two-voter cluster cannot reach quorum when the
foreign peer is unreachable, wedging a previously healthy single
master. Gate raft admission on having been configured with cluster
peers; masters started with a peer list keep the announce-to-join
flow used to grow the cluster.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* master: source SingleMaster from -peers=none, not the peer count
A master started with an empty peer list ends up with a one-peer map
containing itself, and NewMasterServer inferred declared single-master
mode from that count. The master still mints a raft cluster through the
bootstrap path, but AdmitRaftPeer then ignores every later joiner, so a
deployment that grows from one master to several can never form a
quorum. isSingleMasterMode already spells out the declared case
(-peers=none); populate MasterOption.SingleMaster from it instead.
* master: cover clustered raft admission with a real in-memory leader
The previous clustered test had no raft leader and asserted nothing, so
an AdmitRaftPeer that always returned early still passed. Bootstrap a
single-node hashicorp raft over the in-memory transport, wait for
leadership, and assert the leader reports and admits a missing peer.
* master: return the raft admission result and assert committed join
AdmitRaftPeer ran AddVoter on a detached goroutine, so the only observable
state was an in-flight marker deleted on success and failure alike, and the
clustered test could never tell whether the join worked. Admission now
hands back a result channel; the test runs a connected second in-memory
node and asserts the peer sees itself a committed voter.
---------
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* s3api: write the null version's lock metadata to the plain entry
PutObjectRetention and PutObjectLegalHold with versionId=null loaded the
pre-versioning entry but wrote the update under <key>.versions/v_null, so
the acknowledged lock never bound the object it was read from. Resolve the
entry and its write path in one helper that applies the same null-version
rule as getSpecificObjectVersion.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* s3api: cover the null-version lock write path
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* s3api: persist object-lock metadata with an in-place entry update
mkFile rebuilds the target entry from chunks alone, so a lock write on a
plain (null-version) entry discarded attributes, inline content, and the
remote reference. UpdateEntry keeps the whole fetched entry, matching how
the tagging and ACL writers persist metadata.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* release: emit digest-bound provenance and SBOM attestations
Per-platform builds now record SLSA provenance at mode=max and a
BuildKit SBOM, so the builder identity and resolved dependencies are
captured instead of the empty predicate the 4.48 build record carried.
After the multi-arch index is assembled and signed, the merge job
publishes GitHub artifact attestations bound to the merged digest and a
syft SPDX SBOM of the image, giving digest-pinning consumers a
verifiable record on the exact index they pull.
* release: keep unfixed findings in the trivy SARIF upload
ignore-unfixed dropped every finding without an available patch, so the
uploaded report understated the image's exposure. The scan stays
report-only; only the completeness of what is uploaded changes.
* release: generate and attest SBOMs per platform digest
An index-wide syft scan only describes its default platform, so the SBOM
attested to the merged multi-arch digest misrepresented every other
manifest. Scan and attest each platform digest instead, which also drops
the sbom-action upload defaults that raced release-asset permissions and
artifact names across the merge matrix.
* release: resolve per-platform index to its image manifest before syft/cosign
push-by-digest records point at OCI indexes (image + attestation
manifests), so scanning or attesting the index digest picks the
runner's platform and fails outright for the other arches under
set -e. Resolve the non-attestation manifest inside each index and
bind the SBOM attestation to the image digest a same-platform pull
of the merged index actually returns.
---------
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* notification: honor the config prefix in the multi-queue guard
validateOneEnabledQueue checked "<name>.enabled" without the
notification prefix, so the fatal-on-two-queues guard never saw the
real keys and could never fire.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* notification: support named webhook targets under [notification.webhook.<name>]
The webhook queue only accepted one endpoint, so events could not be
routed to different services by path prefix. Each enabled
[notification.webhook.<name>] subsection now spins up its own target —
own filter, http client, retry middleware and worker budget — published
on a per-target topic so every matching endpoint gets each event exactly
once. The flat [notification.webhook] section keeps working as the
unnamed target, and named subsections alone now activate the queue.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* notification: require enabled for the flat webhook target
buildTargets included the flat section whenever an endpoint was set, so a
disabled [notification.webhook] block — which the scaffold ships with a
placeholder endpoint — would start receiving events as soon as any named
target activated the queue.
* notification: resolve named target enabled flags through config
namedTargetPrefixes read the raw map value, which never sees environment
variable overrides; GetBool resolves WEED_* env bindings, so a named
target disabled in the file can now be switched on by
WEED_NOTIFICATION_WEBHOOK_<NAME>_ENABLED.
* notification: cancel the webhook queue only after all workers stop
Each target's router goroutine deferred the shared cancel, so the first
worker to exit tore down the queue context for every other target and
the dead letter logger.
* notification: isolate webhook targets behind per-target queues
SendMessage published to each target's topic inline, so a subscriber
channel that filled up blocked every later target and the filer's
notification path with it. Each target now drains its own inbox through
a forwarder goroutine; SendMessage fails fast on a full inbox, matching
the "throw message" contract the filer already applies to errors.
* notification: name the webhook target in failure and dead letter logs
Every target dead-letters to the same topic, so without the name there
is no way to tell which endpoint failed or needs a replay.
* notification: assert delivery in the named targets test
The test only exercised SendMessage; it now records requests on both
endpoints and checks matching, non-matching, and unrouted keys reach
the right servers exactly once.
* notification: wait for the webhook router before accepting events
* notification: apply backpressure instead of dropping a full webhook inbox
* notification: bound queued and in-flight webhook deliveries per target
* notification: release the pending token when shutdown races the enqueue
A SendMessage that wins the pending select as the queue is canceling
could block forever on a full, undrained inbox.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* notification: stop waiting for a router that failed to start
A router that exits before reaching Running() would leave initialize
blocked on a context nothing had canceled yet.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* notification: route webhook events past saturated targets
A full pending reservation made SendMessage wait on the slow target
before it ever reached the healthy ones after it, coupling every
endpoint's latency to the most congested one. Targets that can take a
reservation now get the event immediately; saturated targets defer to a
second pass that still blocks for a slot, so backpressure is kept and no
event is dropped.
* notification: retry webhook deliveries before dead-lettering
PoisonQueue must wrap the retry middleware: added after it, the poison
middleware intercepts the first handler error, republishes to the dead
letter topic, and returns nil — the retry chain never sees a failure, so
max_retries/backoff settings were dead code and every delivery failed
straight to dead letter on the first attempt.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* s3api: support s3:RequestObjectTagKeys in bucket policies
Condition operators accept the ForAllValues: and ForAnyValue: qualifiers,
and s3:RequestObjectTagKeys holds the keys of the tags taken from
X-Amz-Tagging wherever s3:RequestObjectTag/<key> is set.
CopyObject no longer stores an empty-key tag for an empty segment in
X-Amz-Tagging, so it keeps the key set a policy has checked.
Fixes#7679
* s3: parse tagging header like the tag handlers, not url.ParseQuery
url.ParseQuery stops at 10,000 parameters, so an X-Amz-Tagging header
padded with empty & segments made requestObjectTags return nil: the copy
handler still stored the denied tag while every s3:RequestObjectTag*
condition saw nothing. Scan non-empty segments the way parseTagsHeader
does so the policy sees the same tags the object gets.
* test: gofmt s3tables test files after #11997
The format check fails on these files on master as merged, so every
PR's S3 Tables Format Check goes red until they are re-formatted.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: user.email <chrislusf@users.noreply.github.com>
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* ci: invoke pip via python -m in s3tests clone steps
The self-hosted runner reaches the Clone s3-tests step with the
setup-python toolcache interpreter on PATH but no pip shim, so the
bare 'pip install' exits 127. python -m pip resolves through the
interpreter that is already on PATH, matching the helm_ci workflow.
* ci: invoke pip via python3 -m in s3tests clone steps
The adjacent fix_s3_tests_bucket_conflicts step already runs python3, so
python3 is the interpreter guaranteed on the self-hosted PATH; a missing
python alias would fail the same way the missing pip shim did.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* iceberg maintenance: a merged file with null row ids names no first_row_id
Compaction stamps a merged v3 data file with the _row_id of its first row as
first_row_id (mergedFileFirstRowID). When the bin mixed rows that carry a
materialized _row_id with rows that carry none (a data file with no
first_row_id, as a v2 table's files are after an upgrade to v3, gives null
ids), the null rows then inherit first_row_id + their position in the file:
ids that other rows already hold, so two rows answer to the same row id.
A merged file now names a first_row_id only when its footer shows no null
_row_id in any row group. Otherwise the commit assigns the file a fresh range
from the table's next-row-id: the null rows get new, unused ids and the
materialized ones keep theirs (a materialized value overrides inheritance).
Test: TestMergedFileFirstRowIDOnlyWithoutNullIDs ("materialized first, a null
later" fails without the fix).
* iceberg maintenance: name no first_row_id when the merged file has a null _row_id
mergedFileFirstRowID stamped the merged file with its first row's id
without checking later rows. A null _row_id in a merged file inherits
first_row_id+position, an id another row already holds, so the commit
produced two rows answering to the same row id.
Check the footer's null counts for the _row_id column across row groups
first: any null means the file is committed without a first_row_id, so its
nulls take fresh ids from the table's next-row-id while materialized
values still override inheritance.
* shell: retry filer discovery until master knows a filer
A shell started against a freshly booted cluster (weed mini, the
s3tables/lance test harnesses) can list cluster nodes before the filer's
first heartbeat lands on the master. The empty list left FilerAddress
unset and the next command died with "dial tcp: missing address". Poll
ListClusterNodes until a filer appears or the connect timeout expires.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* shell: discover the filer on first use instead of at startup
The startup retry loop made every weed shell invocation wait out the
connect timeout on clusters that have no filer, even for commands that
only talk to the master. Discovery now runs once, on the first command
that actually needs a filer, keeping the retry-until-registered fix for
the fresh-cluster race without stalling master-only commands.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* shell: stop filer discovery retries on non-transient errors
A master that keeps answering ListClusterNodes with a permanent error
(permission denied, unimplemented) cannot gain a filer by retrying, so
discovery returns instead of polling until the deadline.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Javier de la Torre <jatorre@lakehousebox.com>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: Chris Lu <chris.lu@gmail.com>
* shell: discover the filer on first use instead of at startup
The startup retry loop made every weed shell invocation wait out the
connect timeout on clusters that have no filer, even for commands that
only talk to the master. Discovery now runs once, on the first command
that actually needs a filer, keeping the retry-until-registered fix for
the fresh-cluster race without stalling master-only commands.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* shell: stop filer discovery retries on non-transient errors
A master that keeps answering ListClusterNodes with a permanent error
(permission denied, unimplemented) cannot gain a filer by retrying, so
discovery returns instead of polling until the deadline.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* shell: explain an empty filer list when discovery times out
* shell: retry filer discovery while the address is still empty
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* filer.remote.sync: dispatch an update event by the side of the mount it touches instead of slicing the mount prefix off a path outside it
The daemon subscribes to its mount prefix and to /etc/remote. An update event
whose path lies outside the mount still reaches processUpdateEvent: an update
of the /etc/remote directory entry itself (which fs.meta.load of a mapping dump
re-creates) through the /etc/remote prefix, or a rename across the mount
boundary through the path on the inside. processUpdateEvent then maps both
paths to remote keys by slicing the mount prefix off, which panics with
"slice bounds out of range [20:11]" on a path shorter than the mount and names
a key in the wrong tree on a longer one. The event sits in the replay window,
so every restart dies on it and the persisted offset never moves.
Before dispatch, an update event is now mapped onto the side the mount covers:
a rename into the mount is the creation of the new entry, a rename out of it
the deletion of the old entry's object (kept when the entry is remote-only and
the object holds its only copy), an event with neither path inside is not the
mount's and is dropped, and an update with both paths inside is unchanged.
* filer.remote.sync: extract processCreateEvent
Moves the create branch of eachEntryFunc into a function, matching
processDeleteEvent and processUpdateEvent, so the create path can be
exercised directly in tests. No behavior change.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* filer.remote.sync: upload a create reshaped from a rename into the mount
A file synced under the mount keeps its RemoteEntry across renames. The
rename out deletes the remote object, and the rename back in reshapes to a
create whose inherited stamp still matches the entry, so shouldSendToRemote
skipped the upload and the new key stayed empty. The reshaped create now
sends unconditionally: the stamp describes the old key, not this one.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* filer.remote.sync: a create reshaped from a rename into the mount writes an empty entry as a zero-byte object
* filer.remote.sync: a create reshaped from a rename sends a stamped entry truncated since its sync
A stamped entry still reports remote-only once truncated to zero (the
stamp's RemoteSize stays positive), so the remote-only exemption skipped
it and the new key got no zero-byte object. Gate the exemption on
isDroppedLocalCopy: only an entry whose size still matches its stamp has
its content already at the new key via the rename's remote copy.
* filer.remote.sync: recover a superseded create reshaped from a rename-in
A create reshaped from a rename into the mount whose write is superseded
was skipped like a plain create, but no later event covers this
destination: remote.uncache accepts the rename's inherited RemoteEntry
stamp and drops the local chunks, and the update it logs skips the
upload through isDroppedLocalCopy, while the source mount's sync still
deletes the source object. The destination object is then missing and
the content exists nowhere.
recoverSupersededCreate finishes the event from the entry the filer
holds now: a rewritten entry is left to its own event's upload or
uploaded here on the stamp edge, and a remote-only entry is completed by
copying the object its stamp describes -- resolved through a
pending-rename marker under this mount, or through the filer's mount
mappings to the source mount's remote location.
---------
Co-authored-by: james <james@medable.com>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: chrislusf <chris.lu@gmail.com>
* master: cap parallel vacuums per rack via -maxParallelVacuumPerRack
Hosts that run one volume server per disk under a single rack label could
not bound host-level vacuum I/O: maxParallelVacuumPerServer limits each
process, so every disk on the machine compacted at once. A second quota
map keyed on rack id debits alongside the per-server one, so at most N
volume copies compact in parallel within a rack. Defaults to 0 (no
limit); the flag is wired on master, server, mini and master.follower.
* master: resolve a data node's rack key once per vacuum sweep
A volume server that unregisters mid-sweep loses its parent node, so the
quota credit path calling dn.GetRack() panicked and crashed the master.
Rack names are also local to their data center, so same-named racks in
different centers shared one quota bucket. Each node's rack key is now
captured when the limiters are built and carries the data center id, so
checks, debits and credits never touch the live parent pointer.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* master: reserve vacuum quota per needed copy count, not per copy
Each copy of a volume checked the shared server/rack quota independently,
so several same-rack replicas all passed against one remaining slot and
the debits drove the bucket negative. Needs are now counted per server
and per rack first; a volume whose own copies exceed a cap reserves the
whole bucket once it is free instead of pending forever.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* master.follower: register -maxParallelVacuumPerRack as a flag
It was only a fixed default, so passing the flag on the command line
failed flag parsing.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* master: key the rack vacuum quota by rack node identity
Joining data center and rack names with "/" still let two different racks
share one bucket when the names themselves contained slashes. Key the
limiter by the rack node instead, which is unique per rack regardless of
naming.
* master: keep the tiered-replica vacuum test through the rebase
* master: document 0 as no limit for -maxParallelVacuumPerServer
The clamped per-server need makes a non-positive cap mean "no limit",
matching the new rack flag; say so in the flag help.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* master: key the rack-key cache by data node identity
Node ids are only unique within a rack, so two servers that share an id
in different racks would have reused the first server's cached rack and
escaped the per-rack vacuum cap.
* master: read the per-rack vacuum cap from master.toml
-maxParallelVacuumPerRack becomes the master.max_parallel_vacuum_per_rack
config key, so the new master limit is set the same way as other master
options instead of a flag wired into every command that embeds a master.
* master: let master.max_parallel_vacuum_per_server override the flag
The released -maxParallelVacuumPerServer flag keeps working, but a value
in master.toml now wins when set, matching how master.mdir already
overrides -mdir.
* master: group the vacuum caps under master.vacuum in master.toml
Rename master.max_parallel_vacuum_per_rack to master.vacuum.max_parallel_per_rack
and master.max_parallel_vacuum_per_server to master.vacuum.max_parallel_per_server,
so the keys sit in one subsystem table like master.volume_growth and
master.maintenance.
* master: deprecate -maxParallelVacuumPerServer for master.toml
The flag still works - and is overridden by
master.vacuum.max_parallel_per_server when set - but the help text now
points operators at the config key, matching the codebase style of
help-text-only deprecation.
* topology: read node parent atomically
UnlinkChildNode clears a data node's parent under the rack's lock while
readers like the vacuum sweep's rack capture reach it with no shared
lock, so a torn interface read could crash the master. Store the parent
in an atomic.Pointer and nil-guard the detached walk in
DataNode.GetTopology.
---------
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
A fresh single-filer mini registers its filer only after the ~10s
empty-membership confirm window, and weed shell locates the filer via
the master's listing, so remote.configure sent right after the S3
health check races it. Ports the wait from the enterprise test to the
OSS copy.
* iceberg maintenance: match percent-decoded keys when checking file references
Writers may percent-escape partition directory names in the file paths they
record in manifests (e.g. "part%3D1"), while the S3 layer decodes the key
before storing the object. remove_orphans compared the escaped reference to
the decoded filer path, counted the live file as an orphan, and deleted it
once it passed the age window — readers then got NoSuchKey on the next scan.
Match both spellings of every referenced path in remove_orphans, apply the
same coverage to the kept-file set in expire_snapshots, and delete expired
files at their decoded key so the object the S3 layer stored is the one
removed.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* iceberg maintenance: retry filer lookups at the percent-decoded key
A manifest reference carrying an escaped segment resolves to a filer name
that does not exist, because the S3 layer decoded the key on write. When a
lookup misses and the reference contains escapes, retry once at the decoded
spelling so maintenance can still read the file.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* iceberg maintenance: resolve referenced files verbatim-first, decoded fallback
filer_pb.LookupEntry reports misses as ErrNotFound rather than a gRPC
NotFound, so the escaped-reference retry never fired against a real filer.
Both maintenance reads and expired-snapshot deletes now resolve through one
verbatim-first/decoded-fallback lookup, and expiry refuses to delete a
resolved path outside the table's data directory.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* iceberg maintenance: test expiry of a file stored at a literal percent key
* iceberg maintenance: resolve referenced files in one filer lookup
lookupFileByIcebergPath re-fetched the entry resolveFilerPath had just
looked up, doubling the RPC per referenced file during compaction and
expiration. The resolver now returns the entry alongside the path so a
hit costs one lookup and a decoded fallback two.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* iceberg maintenance: keep every decoded spelling of a referenced path
The kept-reference sets recorded a path plus a single decode of it, but
lookupResolvedEntry decodes recursively until the spelling is stable. A
doubly-escaped reference could resolve to a spelling absent from the set
and be deleted while still referenced. addFilerPathSpellings now walks
the same decode chain the resolver does.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* iceberg maintenance: keep every decoded spelling of a referenced path
* iceberg maintenance: test expiry keeps a doubly-escaped kept reference
* iceberg maintenance: clean decoded spellings in the referenced-path set
* iceberg maintenance: test expiry keeps a kept reference resolved via decoded parent segment
* iceberg maintenance: decode only the reference in the spelling set
Decoding the composed filer path also decoded the table directory, so a
table at a literal percent-escaped path produced a kept set missing the
spelling the resolver returns, and expiry could delete a live file.
* test: gofmt the s3tables cluster helpers
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* filer.conf: merge fsync on presence like worm
fsync is a pure preference, so OR-merging it meant a nested rule could
never turn a parent's fsync off. Make PathConf.fsync optional and merge
it most-specific-wins, matching worm: an unset rule inherits, an explicit
value overrides in either direction. readOnly and disableChunkDeletion
stay OR'ed — they are fail-closed locks quota enforcement depends on.
Pre-version-2 configs were written with EmitUnpopulated, so every rule
carries an explicit fsync=false that meant nothing; it is dropped to
unset on load so it does not read as an opt-out.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* filer.conf: test fsync override survives a save/reload round trip
---------
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* operation: euthanize pooled upload conns that stall far below a fast host
A pooled keep-alive conn can wedge and drain the request body at a trickle
while fresh conns to the same volume run at full rate; every later upload
routed onto it stalls for tens of seconds, indefinitely. Track per-host
healthy-rate uploads (>=1MiB/s on a >=256KiB body) and, on a host that was
fast within the last minute, cancel an upload still in flight past
max(10s, size/256KiB/s): the transport discards the wedged conn and the
existing retry dials a fresh one. A watchdog kill also clears the fast mark
so the retried attempt runs unbounded, and hosts that never prove fast keep
today's unbounded behavior - a chronically slow link is never penalized.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* operation: keep the stall watchdog armed through the response read
uploadDone was set as soon as Do() returned, so a conn that accepted the
request body then wedged on the reply escaped the watchdog and could
park the upload for hours — the same failure this watchdog exists to
kill. Deferring the done-store keeps the timer live until the function
returns, so a stalled io.ReadAll is cancelled and retried like a
stalled write. The redundant post_err recheck goes with it.
* operation: pin watchdog coverage of a stalled response read
* operation: disarm the stall watchdog for an upload's remaining attempts
Once the watchdog has cancelled an attempt, a concurrent healthy upload
to the same host can restore the fast mark during the retry backoff.
The re-armed watchdog would then cancel the retry too, and a large
upload that merely exceeds the floor could burn every attempt. Track
the watchdog kill on uploadStatusError and carry a
stallWatchdogDisarmed flag through retriedUploadData so the rest of
that upload's attempts run unbounded; other uploads to the host keep
their watchdog.
* operation: pin watchdog disarm on retry, fresh-dial, and slow-upload cases
* operation: drop the send-stall watchdog once the request body drains
A wedged pooled conn is only blameworthy while it is still trickling the
request body; once the body is on the wire the wait is server-side commit,
which the transport's response header timeout already bounds. Cancelling
then would fail a healthy write whose replication or fsync is simply slow.
Track body EOF through a wrapping reader, gate the send-side timer on it,
and re-arm the watchdog only around the response-body read, which the
header timeout does not cover.
* operation: prove a host fast only on a verified upload
A quick rejection moves bytes no faster than the wire did, so mark the
host from the body-send duration and only after the response checks out.
* operation: count an abandoned request body as sent
The transport closes a request body it never finished draining when the
server answers early, so EOF alone under-reports the send boundary and
the stall watchdog could still blame the conn during the reply wait.
* operation: scope the body-sent mark to the current attempt
A lagging Close or EOF from the first attempt's body could stamp the
shared mark after the inner retry reset it, disarming the watchdog for a
send that never happened; each attempt now points the watchdog at its
own sent timestamp.
* operation: drop only the fast mark the stall timer observed
A concurrent upload can re-prove the host while a stalled attempt is in
flight; deleting the key outright would erase that newer mark, so the
watchdog now removes only the value it armed against.
* operation: keep the stall disarm on a per-call upload option
retriedUploadData marked the caller's UploadOption, so a caller reusing it
across UploadData calls would start the next upload with the watchdog off
and lose the cancellation this adds.
* test: stall watchdog retry must fail if the conn gets cancelled
The fake retry slept through any cancellation, so a re-armed watchdog
firing mid-retry still let the upload succeed — the guard was untested.
Park on the request context like a real wedged conn so a re-armed
watchdog cancels the attempt and fails the test.
* operation: prove the fast-host mark with elapsed-to-response
Body-drain time only measures when the transport consumed the reader:
kernel socket buffers can swallow a small body instantly on a slow link,
inflating the rate and marking the host fast. The next upload then gets
its conn cancelled mid-send despite making progress. Elapsed-to-response
bounds remote speed from below — the server cannot answer before the
body lands — so the mark can only under-mark, never over-mark.
* operation: scale the stall budget off the host's measured rate
Fixed bps limits fit no link well: 256KiB/s never fires on a slow link
and waits far too long on a fast one. Record the wire rate of each
sizeable upload and let a later upload run for slowFactor times the
body's span at that rate — a wedged conn shows as an order-of-magnitude
drop on any link speed. The recorded rate uses the wire body size so a
pre-compressed payload no longer inflates the mark.
---------
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* filer.backup: process metadata events through the concurrent processor
Each event was handled inline on the subscription loop, so one
fetch+write at a time capped the stream at a few entries per second and
a busy source kept building lag. Route events through the same
MetadataProcessor filer.sync uses: it fans jobs out behind -concurrency
while serializing same-path and subtree-barrier conflicts, and the
checkpoint now advances on the processor's contiguous watermark via
GetResumeTsNs, with Resubscribe replaying anything a failure pins.
-disableErrorRetry keeps its log-and-skip meaning by returning nil after
logging, since the processor would otherwise pin the checkpoint forever.
* filer.backup: drain in-flight jobs before the stream retry loop reconnects
FollowMetadata returns as soon as the stream ends while admitted jobs keep writing; the next doFilerBackup builds a fresh processor that cannot order its replayed writes against them, so an older upload could finish last and clobber a newer file.
* filer.backup: checkpoint the completed watermark on a ticker and at stream end
The offset callback only fires while events arrive, so work draining on a quiet stream stayed uncheckpointed until the next event and a restart recopied it. Persist the processed watermark every 3s when it advances, and once more after the post-stream drain.
* filer.backup: bound the checkpoint wait before reconnecting
setOffset has no context, so a stalled KvPut would hold persistDone and block the retry loop forever. The persist goroutine now performs the final write itself on stopPersist, and the caller waits at most 30s.
* filer.backup: carry the processed position across retry attempts
A follow attempt that ended with a pinned failure recomputed startFrom on retry, so a -timeAgo window could move past the failed event and skip it. doFilerBackup now returns the failure pin (or processed watermark) and the retry loop clamps startFrom to it.
* filer.backup: resume retries at the previous attempt's pinned position
The retry loop threaded resumeTsNs through but never consulted it, so a
-timeAgo run recomputed startFrom as now-minus-the-window on every attempt
— a failed event older than the window was skipped on reconnect. Use the
pinned position when one exists; -timeAgo and the persisted checkpoint
only choose the first starting point.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* filer.backup: resume below the failure pin so the failed event replays
The metadata reader treats the resume position as exclusive — events at
or below it are skipped — so resuming at the failed event's own timestamp
would skip it. Back one tick off the pin.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* filer.backup: label the progress line as the received-event rate
The counter increments when a job is admitted, not when its copy
finishes, so the old wording showed backup throughput during a burst of
slow copies while the processed watermark had not moved.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* replication: bound the offset checkpoint RPC
setOffset called KvPut on the background context, so a stalled write
could pin a caller's drain or persistence loop indefinitely.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* filer.backup: keep the resume pin when filer-source setup fails
A retried attempt carries resumeTsNs so the failed event replays from the
same position, but filer-source and SSE initialization failures returned 0,
so the next attempt recomputed -timeAgo and could move the window past the
pinned event.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* replication: serialize checkpoint writes per key
A checkpoint write can outlive its caller's wait and land beside the next
subscription attempt's writes, moving saved progress backward. Serialize
writes per key inside setOffset and drop any value older than one already
written from this process.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* filer.backup: gofmt
* replication: serialize checkpoint writes per filer and key
A single process-wide lock made a stalled checkpoint RPC block every
stream's checkpoint, including the opposite direction of a bidirectional
filer.sync. Keeping the last-written watermark per persisted key also let
one stream's offset shadow another stream checkpointing the same key to
a different target filer. Lock and track progress per (filer, key).
* replication: keep the resume position behind a cleared failure pin
A redelivery that clears failedTs lets the processed watermark advance
past the earliest failure, but events dropped while the processor was
stopped — possibly sharing the failure's timestamp — have to replay.
Latched earliestFailedTsNs floors the resume position one tick below
that failure so reconnects still cover them.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* replication: persist floored checkpoints through the write ordering
setOffset drops a value older than the last written, which also swallows
a deliberate regression: the resume floor, an operator-seeded offset, or
a snapshot high-water mark. allowRegress marks those callers so the
checkpoint can move backward when a failure forces replay.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* replication: order checkpoint writes by issuance, not value
The value-based stale guard could never fire — every caller passed
allowRegress because floors and seeds are deliberate regressions — and
ordering by value cannot tell a stale high watermark apart from real
progress: a write that outlives its attempt could still land after a
floored resume and skip events owed a replay. Order by issuance instead:
a write issued before the last one to land under the key is dropped, so
late writes from a superseded attempt can no longer move the checkpoint
in either direction while deliberate regressions still apply.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* replication: keep the resume position one tick below the frontier
* remote.sync: resume and persist through the floored watermark
* test: concurrent metadata jobs fan out up to the limit
Distinct-path events must all be in flight before any finishes — a
serial loop could only ever have one running — while the limit still
bounds the pool. Pins the property behind the backup-throughput fix.
---------
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
The rocksdb release image fails to compile: grocksdb v1.11.1 calls
C.rocksdb_block_based_options_set_block_align,
C.rocksdb_block_based_options_set_index_block_search_type, and
C.rocksdb_block_based_options_set_separate_key_value_in_data_block, all
of which only exist in RocksDB v11.x headers. grocksdb v1.11.1's own
build script pins 11.1.2, so move every default from v10.10.1 to
v11.1.2 to match.
Generated with [Devin](https://devin.ai)
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Self-hosted runners do not ship fuse3, so the setuid repair step
failed with 'no fusermount3 in PATH' before the FUSE jobs could run.
Install the package instead of failing outright.
New upstream releases build for x86-64-v3 (AVX2) - ClickHouse since
26.5, plus the DuckDB iceberg extension and pylance wheels - so on a
runner whose CPU lacks AVX2 the container dies on SIGILL before the
test reaches any SeaweedFS code. Probe each third-party image first
and skip on the illegal-instruction exit instead of failing; other
errors still surface normally.
A zero local commit time conflated 'no local state' with 'state at the
epoch', so a missing entry was skipped whenever the sender reported no
commit time and the entry mtime was at or before the epoch. Only hold
the copy off when a local entry actually exists.
The registry container is started with --restart=always but never
removed during teardown, so once a machine runs the job, every later
run on that machine fails with a container name conflict. Reuse the
running container when it exists and recreate it only when missing or
broken.
Some self-hosted runners (e.g. seaweedfs-server03) also lack the
Docker Compose plugin, so install it via docker/setup-compose-action,
which is a no-op where compose is already available.
* volume.fsck: queue per-volume orphan deletes through the shared purge pool
The worker pool added for -reallyDeleteFilerEntries only covered entries
whose volume was missing entirely; the per-volume check that finds filer
entries pointing at absent needles still issued one synchronous HTTP
DELETE per orphan, which is the bulk of the work the report measured.
One pool now serves both purge phases and is drained before the empty-
directory sweep.
* volume.fsck: keep hard link ids in the filer fid file
Orphans queued by the per-volume check must hash into the pool by their
hard link id just like the missing-volume path, so two paths sharing an
id still serialize on one worker. The fid file now stores the id beside
each path and readFilerFileIdFile hands it to the caller.
* volume.fsck: cover orphan purge queuing and hard link id round-trip
* volume.fsck: fail on a truncated hard link id in the fid file
A .fid record cut short inside the hard link id length was emitted with a
nil id, so its purge could hash to a different worker than the entries it
shares the link with. Read the length strictly like the header and path.
* volume.fsck: drain the save consumer before returning a root genFn error
The early return left the save goroutine blocked on outputChan while the
caller ran finishPurge, so a late queuePurge could send on a closed
channel and panic.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* storage: skip EC shard-size encoding tests when disk space is low
TestCalculateExpectedShardSizeWithRealEncoding writes a .dat file plus
14 EC shards per subtest (~1.3GB peak for the 512MB case) into the temp
dir. Shared CI runners can run low on space while other test packages
write to /tmp in parallel, and the encode then dies with ENOSPC
mid-write, surfacing as a test failure. Check free space up front and
skip the subtest instead.
---------
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* s3: url-encode ListMultipartUploads results for encoding-type=url
The response echoed EncodingType=url but returned raw upload keys, and
XML 1.0 cannot carry control characters, so the encoder replaced them
with U+FFFD. The listed key then matched no upload: AbortMultipartUpload
with it answered 404 NoSuchUpload, leaving such uploads uncleanable.
Encode the key-named elements AWS encodes for this API (Key, Prefix,
Delimiter, KeyMarker, NextKeyMarker) the way ListObjects encodes its
keys when encoding-type=url is requested. Upload ids stay opaque, and
responses without encoding-type=url are unchanged.
Fixes#11991
* s3: echo EncodingType only when encoding-type=url is honored
A request with an unrecognized encoding-type echoed the value back in
<EncodingType> while the keys stayed raw, matching neither AWS (which
omits the element unless url encoding applies) nor the ListObjects
handlers, which gate the echo on the parsed url flag.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: chrislusf <chrislusf@users.noreply.github.com>
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Unformatted test files leave the S3 Tables Format Check red on every
branch until they are re-formatted.
Generated with [Devin](https://devin.ai)
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test: raise mini-server startup wait from 30s to 90s in Makefiles
Self-hosted/persistent runners under load regularly need >30s for
weed mini (master+volume+filer+s3+iceberg+lance+webdav+worker) to
become healthy; hosted runners start in <10s. Siblings already wait
90s (s3/checksum, s3/cors). Fixes the S3 Filer Group first-run flake
on busy self-hosted runners.
* test: align Makefile startup guards and progress totals with 90s bound
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
The workflow relied on a Go toolchain preinstalled on the runner
image; self-hosted runners without setup-go fail at autobuild with
'go: executable file not found'.
* ec: measure shard placement against replica-placement caps
ecShardPlacementViolations reports where a volume's newest-generation
shard placement exceeds the per-node and per-rack caps a
-shardReplicaPlacement requests, using the same digit mapping the
balance planner enforces (SameRackCount = shards/node, DiffRackCount =
shards/rack, dc-qualified rack keys).
* ec.encode: -strictPlacement keeps originals when placement breaches caps
-shardReplicaPlacement tells the balancer the requested per-node and
per-rack shard limits, but nothing verifies the outcome. With
-strictPlacement the pre-delete shard verification also measures the
final placement against those caps: a volume still over the node or
rack limit aborts before its source is deleted, so the encode only
completes when the requested failure-domain spread is actually met.
* ec.encode: cover the strict-placement delete gate with a test
The gate in verifyEcShardsBeforeDelete now runs against a stubbed
topology: a compliant spread passes while an over-cap placement is
refused before source deletion. The retry knobs become package vars so
the test does not wait out the heartbeat-propagation backoff.
* ec.encode: enforce -shardReplicaPlacement without a flag
Drop the -strictPlacement option: the requested per-node and per-rack
caps are always requirements at the pre-delete verification, so the
encode only completes when the placement actually honors them.
* iam: GetUserPolicy returns NoSuchEntity for a policy name the user does not have
When the user has stored inline policies, an unknown name fell through to the
policy rebuilt from the user's actions, so a deleted policy still read back.
The standalone IAM server already returns NoSuchEntity here.
* iam: return service failure when listing inline policies fails
A ListUserInlinePolicies error in GetUserPolicy fell through to
action-based reconstruction, masking a store failure as a policy
document. Propagate it as ServiceFailureException instead, matching
the standalone iam server and the other callers in this file.
* iam: return service failure when the inline policy store read fails
---------
Co-authored-by: chrislusf <chrislusf@users.noreply.github.com>
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* s3: gate POST Object acl fields behind PutObjectAcl
The POST Object handler promotes acl and x-amz-grant-* form fields onto
the upload's ACL headers, and resolveFileMode honors the promoted canned
ACL. The PUT path only applies such headers after an s3:PutObjectAcl
grant; mirror that on POST by running the shared per-key authorization
with the WriteAcp action whenever the request carries ACL headers.
* s3: cover the POST Object acl gate
Signed POST requests carrying an acl form field are denied for a writer
without the WriteAcp action and for a writer a bucket policy denies
s3:PutObjectAcl on the target key; the authorization boundary is
exercised directly as well.
* s3: cover x-amz-grant-* POST fields and the no-ACL case
* s3: encode POST policy conditions as JSON in the acl-gate test
* master: cap writable volumes per collection via -maxWritableVolumes
Some deployments serialize disk writes (e.g. an Unraid array where
concurrent writes to multiple disks collapse throughput). With more
than one writable volume in a collection, the master's weighted pick
spreads assignments over all of them, so several disks get written at
once.
-maxWritableVolumes=N narrows the pick candidates to the N fullest
writable volumes, so a collection's writes concentrate on one (or few)
volumes at a time. 0 keeps the current spread-everywhere behavior.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* master: give master.follower a maxWritableVolumes default
* master: cap writable volumes inside the requested location
* master: pick the capped volumes without sorting the writable list
* master: assert the capped candidate set instead of random pick outcomes
* master: describe -maxWritableVolumes as a per-layout cap
---------
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Follow-up to #11985/#11986: setup-go/setup-java/setup-python steps
added or restructured since (e.g. by #11981's shared-artifact layout)
were missed. On self-hosted runners setup-go restores/saves a ~5GB
module cache archive per job — pure waste when the module cache is
already on local disk — and the restore blocks the job for minutes.
* pb: share the stale-socket tolerant unix listener
ServeGrpcOnLocalSocket already treats the local socket as optional; pull the remove-and-listen block out so the filer, s3, and sftp http sockets can use it too.
* command: keep filer, s3, and sftp up when the local socket cannot bind
A stale socket file owned by another uid (e.g. left in /tmp by a containerized job) made os.Remove fail with EPERM and the Fatalf took the whole server down. The gRPC local socket already treats the unix listener as optional; do the same for the http/sftp sockets.
* filer: clear shared localSocket flag when the unix socket cannot bind
s3.localFilerSocket and (in weed server) the sftp local socket share the
filer's -localSocket flag pointer. If the filer cannot bind the socket it
now keeps running without it, so consumers still holding the configured
path would dial a dead or foreign-owned socket.
* command: keep sftp on its own localSocket flag under weed server
sftpOptions.localSocket was overwritten with the filer's -localSocket
flag, so -sftp.localSocket was ignored and an explicit filer socket path
made the sftp server try to bind the same socket the filer already held.
* s3: fall back to TCP when the filer local socket cannot be dialed
The embedded S3 server copies the filer's -localSocket flag before the filer tries to bind it, so a socket that never comes up left every filer HTTP request dialing a dead path. Try the unix socket first and fall back to the request's TCP address on dial failure.
* filer: keep the shared localSocket flag read-only after startup
Clearing *fo.localSocket raced with the embedded s3 goroutine reading the same flag pointer and could not help a reader that had already copied it. The s3 filer client's TCP fallback covers the unbound-socket case without mutating shared flag state.
server: skip default local unix sockets that cannot be bound
weed server fatally exited during S3, filer, or SFTP startup when the default /tmp/seaweedfs-<svc>-<port>.sock path could not be removed or bound. On shared hosts (e.g. CI runners where another user's weed left a socket in the sticky /tmp directory) a stale socket file turns into 'operation not permitted' and kills the whole server, even though the TCP listeners are unaffected. Warn and skip the unix socket listener instead, matching how the gRPC local socket already degrades. An explicitly configured -localSocket path still fails fast, since a co-located S3 server may be configured to dial the filer through it.
* volume: also listen on localhost like other components
Master, filer and s3 bind both the detected host ip and localhost via
NewIpAndLocalListeners, but the volume server used a plain NewListener on
bindIp, so a volume bound to the host ip never answers on loopback. Any
tooling that polls localhost (e.g. the java test workflow's readiness
curl on localhost:8080/status) then waits out its full timeout even
though the volume is up and registered with the master.
Use NewIpAndLocalListeners for the volume grpc, public and cluster http
listeners, and add a small multiHttpdownServer composite so graceful
shutdown still covers both listeners.
* volume: stop and wait dual listeners concurrently
A serial multiHttpdownServer.Stop let the localhost listener keep
accepting requests while the bound-ip listener drained, and a serial
Wait hid a localhost serving failure until the bound-ip listener ended.
Run both waits and both stops in parallel: Wait reports the first error
promptly, Stop closes every listener when shutdown begins.
* tests: pin weed mini admin ports and disable unused aux services
Every weed mini that does not configure them still binds the fixed
default ports for the admin server (12535 + derived 22535 gRPC), its
legacy compat listener (23646 + 33646), WebDAV (7333), Lance (9101) and
the Iceberg REST catalog (8181). Sequential or concurrent minis on one
host then collide on them: the port check warns and shifts, and the
admin's late bind can land on a stolen ephemeral port, adding noise and
flakiness to test logs (e.g. run 40017181958, job 120659267479).
Add testutil.MiniAdminArgs to pin the admin server to an allocated
mini-style port (HTTP + port+10000 gRPC, both below the Linux ephemeral
port floor since admin binds late) and disable the legacy compat
listener. Every test that spawns weed mini now also disables the aux
services it does not use: WebDAV, Lance and the Iceberg REST endpoint
get port=0 unless the test actually exercises them, in which case their
ports stay pinned as before.
* tests: take mini admin ports from the below-ephemeral range
* tests: reuse MiniAdminArgs for the remote-s3 fetch mini
* tests: fully disable webdav and co-allocate mini admin ports
- -webdav.port=0 only makes the service bind a random port; add
-webdav=false so minis that do not test WebDAV skip it entirely.
- Spark, RisingWave and s3 Spark launchers allocated the admin port in
a second MustFreeMiniPort call, which could pick a port already
handed out by the first batch. Fold 'Admin' into the same
MustFreeMiniPorts call.
- test/fuse_integration is a separate Go module and cannot import
testutil; allocate the admin ports with its local freePort helper
instead.
* test/fuse: derive the mini admin gRPC port from its HTTP port
Two independent freePort calls could return the same port, making admin.port equal admin.port.grpc (or collide with the filer port). Allocate once and derive the gRPC port with the +10000 convention.
* test/fuse: pin the mini admin port in the fuse_test framework copy
The fuse_test package framework spawns weed mini too; without a pinned admin port it still binds the fixed defaults 12535/23646 and collides with other minis.
* tests: drop the redundant -webdav.port=0 flag
-webdav=false alone disables the service; the port flag is dead config.
* shell: retry filer discovery until a filer registers
When -filer is not given, the shell discovers a filer with a single
ListClusterNodes call right after connecting to the master. A filer can
take several seconds to register after its port reports ready, so the
list routinely comes back empty and the shell proceeds with an empty
FilerAddress. Commands then fail with "dial tcp: missing address", which
is how the s3tables Dremio integration test flaked in CI.
Treat an empty list as retryable and keep polling ListClusterNodes until
a filer appears or -connectTimeout expires. On timeout, print a clear
"no filer registered" notice instead of failing later inside a command;
master-only shell commands keep working either way.
* shell: report the last filer discovery attempt failure
A successful empty ListClusterNodes followed by a master outage left the
diagnostic claiming no filer registered, hiding the real RPC failure.
Track the outcome of the latest attempt instead: if the final list call
failed, report that error; only when the last response was genuinely
empty report that no filer registered within the timeout.
* shell: test filer discovery retries against a fake master
Extract the discovery loop into listFilers and cover the retry behavior
with a fake master_pb.SeaweedServer: an empty cluster turning populated,
an empty cluster exhausting the window without error, the last attempt's
RPC failure being reported, and master-only commands still running after
the discovery timeout.
* test/fuse: fix cluster startup in the fuse test framework
weed volume Fatalfs when -dir does not exist, so every cluster setup
failed its volume wait after 30s. The mount also exited on the unknown
-cacheSizeMB flag (it is -cacheCapacityMB now).
* test/fuse: manage cluster processes and allocate free service ports
Each service now runs through a managed process that captures its log and
reports an early exit from the port wait, instead of a bare 30s timeout.
Ports are allocated free pairs (http + grpc sibling) rather than fixed
19333/18080/18888 so the framework no longer collides with weed mini's
fixed ports or another framework instance.
* test/fuse: bail out of the mount wait when the mount process exits early
The port waits already report an early exit with the captured log tail; the
mount wait still polled the device id for the full timeout, so a mount that
died at startup (bad flag, missing filer) surfaced as a bare 30s timeout
with no diagnosis. Check the managed process the same way and include its
log tail.
* test/fuse: deduplicate service ports handed out by freeServicePort
A probed port is released before the service binds it, so the three
sequential allocations for master, volume and filer could all return the
same still-unbound port and leave later services dying at startup. Track
handed-out ports so each allocation is unique.
* test/fuse: allocate service ports in Setup after the external-mount branch
* test/fuse: verify service readiness through its own /healthz endpoint
* test/fuse: bound the readiness probe so a silent listener cannot hang setup
* test/fuse: only accept a ready response while the service process still runs
* test/fuse: include the service log tail on readiness timeout
A startup that stalls instead of exiting still failed with a bare
"not ready within timeout" — the same opaque message that made the
original flake hard to diagnose. The process-exit path already tails
the captured service log; the timeout path now does too.
* test/fuse: report pjdfstest category results without failing the suite
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* ci: allow allow_other mounts in posix-compliance
The job mounts with -allowOthers but never sets user_allow_other in
/etc/fuse.conf, so fusermount rejects the option and the mount exits
before the suite runs. The other FUSE workflows set it; do the same.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test/fuse: keep the standalone-framework service logs on failure
Cleanup removed the temp dir holding master/volume/filer/mount logs, so
a CI failure left nothing to diagnose. Dump the tails on failure and
copy them under /tmp/seaweedfs-fuse-logs/standalone/ for the artifact
upload, next to the mini framework's logs.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test/fuse: gate setup on a real write through the mount
Mount readiness only proves the daemon is serving; a fresh cluster still
cannot assign a file id until a volume heartbeats in and a grow round
finishes, so the first ~10-30s of writes surface as EIO on close. Probe
with a real write/read/remove before handing the mount to the suite.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test/fuse: gate the mini framework setup on a real write too
Same cold-start window as the standalone framework: mini's volume has to
register and grow before a write can be assigned. Keep the two
frameworks consistent so later suites do not flake on the same stall.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test/fuse: save each framework's CI logs under its test name
Fixed-name mini.log/mount.log in /tmp/seaweedfs-fuse-logs let a later
framework instance overwrite an earlier test's logs, so an uploaded
artifact could describe a passing run instead of the failure under
investigation. Apply to both framework copies since they share the
directory.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test/fuse: bound each waitForWritable probe attempt
A wedged mount can hold a file op open past the 90s deadline, so the
loop could not check the deadline or a mount exit and Setup hung until
the overall test timeout. Each attempt now runs off the loop and is
abandoned when the remaining time runs out. Same fix in both framework
copies.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test/fuse: keep one write probe in flight until it finishes or the deadline
---------
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: chrislusf <chris.lu@seaweedfs.com>
The EC integration tests failed in CI with "raft.Server: Not current
leader" on every assign for tens of seconds — longer than any goraft
election takes. A stale weed process on the self-hosted runner was
still holding 9333, so each test's fresh master died at bind time and
the tests kept talking to a raft non-leader. Only the 9333 cluster
(startSeaweedFSCluster) failed; tests on 9334-9338 all passed.
Allocate mini-safe ports (http + grpc+10000 both verified free) for the
master and volume servers instead of hardcoding 9333/8080-8085, wait
for the master to actually report raft leadership via /cluster/status
before issuing raft-gated RPCs, and make waitForServer a real TCP dial —
grpc.NewClient never connects, so the old check always passed
instantly.
* helm: fix bucket hook treating an existing bucket as missing
The skip check pipes awk into `grep -q`, which exits on the first match and
can leave awk writing into a closed pipe. awk then dies of SIGPIPE and
`set -o pipefail` makes the pipeline report that, so a bucket that exists
reads as missing: the hook runs `s3.bucket.create`, it errors with "already
exists", and `sh -ec` aborts the hook - failing the release.
Match in a single awk instead, which reads to EOF and cannot be killed by a
closing reader.
* helm: force string compare in bucket hook existence check
---------
Co-authored-by: Chris Lu <chris.lu@gmail.com>
cargo registry/target and docker-image caches are already local on a
persistent self-hosted runner; the save/restore pass is pure overhead
(the setup-go cache-save stall on awscli-tests was the same pattern).
Gate every actions/cache step on runner.environment == 'github-hosted'.
Generated with [Devin](https://devin.ai)
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* ci: skip setup-go cache on self-hosted runners
The module cache is already local on a persistent self-hosted runner;
archiving ~/go/pkg/mod into cache.tzst at every job's end is pure cost
and can stall (a ~4-minute zstdmt hang was observed on awscli-tests).
runner.environment gates the cache to GitHub-hosted runners only.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* ci: gate the remaining setup-go caches too
The first pass only covered go-version-file steps; go-version steps in
nine workflows (including awscli-tests' own file) kept uploading the
cache archive on self-hosted runners.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* ci: cache docker build layers for container_dev
The Go-build image rebuilds its layers on every run; type=gha backend
caching reuses them across runs and platforms.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* ci: pin container build to the pushed sha under layer caching
Dockerfile.go_build clones master inside the image; a cache-from layer
would otherwise keep serving the previous commit's clone and publish
stale code under the dev tag. Passing BRANCH=github.sha makes the clone
layer's key commit-specific while toolchain layers still hit.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* ci: pass BRANCH via build-args so the per-commit cache key reaches buildx
docker/build-push-action has no `args` input — the BRANCH override was
silently dropped, leaving the build arg at its `master` default and the
clone/checkout layers serving a stale commit under the gha cache.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* ci: build test binaries once, share them across shard jobs
Each of the 6 rust-volume-go-tests shards, the rust-integration-tests
job, and the 6 volume-server-integration-tests shards rebuilt the same
weed and weed-volume binaries — 13 duplicate builds per run. A
build-test-binaries / build-weed job now compiles them once under the
existing cargo cache and publishes them as artifacts; the consumers
download and chmod them instead of building.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* ci: preserve binary paths through the artifact round-trip
upload-artifact stores single files flat, so weed/weed would come back
at the workspace root and the consumers' expected paths break. Pack the
binaries in a tar before upload and extract on download.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* ci: extract the shared test binaries in the rust go-test shards
The download step leaves rust-test-binaries.tar at the workspace root;
without untarring it the chmod fails on missing files and every shard
fails before running a test.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* ci: cancel superseded workflow runs on PR pushes
PR-triggered workflows without a concurrency group keep running (and
holding runner slots) after a force-push or quick follow-up commit
supersedes them. Add the same group/cancel-in-progress block the go
workflow already uses, keyed on head_ref, to the remaining PR workflows.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* ci: key concurrency groups on PR number or ref, not head_ref
github.head_ref is empty on tag pushes and manual runs (every run then
shares one group and cancels the last) and collides across forks when
two PRs share a branch name. pull_request.number is unique per PR;
github.ref covers push/tag/dispatch. Applied to all workflows using the
pattern, including pre-existing ones.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* ci: restrict cancel-in-progress to PR and branch-push runs
* ci: serialize container_latest manual runs by destination tag
Two workflow_dispatch runs sharing an image_tag could otherwise build
concurrently: their per-arch images publish to the same platform tags,
so one run's create-manifest job can combine platforms from different
source refs. Group dispatch runs by the destination tag (with
cancel-in-progress already disabled for non-PR/non-branch events) so
same-tag publishes queue instead of interleaving.
* ci: serialize manual publish runs by destination tag in remaining container workflows
* ci: group foundationdb release dispatch runs by release tag
A manual run now shares the tag-push concurrency group for the same
release, so a re-publish of a tag queues behind (or ahead of) the push
run instead of racing it on the same image tag.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* ci: serialize foundationdb_version dispatch runs by destination tag
Two manual runs resolving to the same docker tag now share a concurrency
group (explicit image_tag, else the computed
foundationdb_<fdb>_seaweedfs_<ref> default), so they queue instead of
racing pushes to the same tag.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* ci: drop unsupported toLower() from foundationdb_version concurrency group
* ci: key rocksdb_version dispatch group on its published tag
* ci: queue publish runs via turnstyle instead of cancel-free shared groups
* Revert "ci: queue publish runs via turnstyle instead of cancel-free shared groups"
This reverts commit 039148f8d5.
* ci: queue same-tag publish runs instead of replacing the pending one
Dispatch/tag-only workflows can never cancel in progress, so opt their
concurrency groups into queue: max — up to 100 same-tag publishes wait
in line rather than each new dispatch cancelling the pending one.
---------
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
The separate master/volume/filer processes in framework.go bound fixed
ports, discarded child output, and never noticed an early process exit,
so a slow or dead volume server burned the whole 30s timeout with no
logs. Switch to the same weed-mini approach the fuse_test package
framework already uses: dynamic filer ports, child output captured to
log files, and readiness waits that bail out the moment the process
dies. The external-mount path (TEST_MOUNT_POINT) is unchanged.
The posix-compliance workflow mounts with -allowOthers, but fusermount
refuses allow_other unless user_allow_other is set in /etc/fuse.conf,
mirroring the fuse-integration workflow FUSE setup.
* s3: end a catalog-vended session with the access key it was vended for
A table credential the Iceberg catalog vends is a self-contained STS session:
it stays valid until it expires (an hour by default) even after the access key
that loaded the table is deleted or deactivated, or its identity is removed
or disabled. An operator revoking a leaked key cannot cut off the table
credentials that key already obtained.
Sessions are stateless JWTs, so there is no revocation list to consult. This
change follows the same pattern as role sessions, which are already refused
once their role's ID no longer matches (checkSessionRoleBinding): a vended
session carries the access key it was vended for (src_ak, recorded by the
previous change), and validateSTSSessionToken checks it against the identity
table held in memory. A session of provider iceberg-catalog that carries a
source key is refused (InvalidAccessKeyId, as for an unknown key) once its
identity is gone or disabled, or the key is no longer one of that identity's
active, unexpired credentials.
Scope:
- Only catalog-vended sessions are checked; web-identity, AssumeRole and
every other STS session are validated as before.
- The vendor binds a session to the caller's key only when the caller is a
stored identity holding that key. A caller that signed with another STS
session's key, an OIDC user, or a request that recorded no key gets an
unbound session, as today; binding it would refuse every request made with
it. A session minted before this change carries no key and is unbound.
- The check is one map lookup under the read lock and a scan of one
identity's keys (about 11 ns, 0 allocations, with 1,000 identities).
Tests (s3api_iceberg_vended_session_test.go) mint real sessions through the
catalog vendor and validate them the way a data-path request is validated:
deactivating, expiring or deleting key A ends A's session while the session
vended through key B on the same identity keeps working; disabling or
deleting the identity ends it; callers that are not a stored identity holding
the key get unbound sessions that keep working; other STS sessions are not
checked. The first two tests and the third fail without this change.
* s3: keep the vended-session key binding when the key dies before vend
vendedSessionKeyLive gated the binding, so a key revoked between the
request's authentication and the vend skipped it and minted an unbound
session that stayed usable until it expired. Every catalog auth path
pairs the identity name in context with the key that authenticated, so
a stored-identity principal always means the request signed with that
identity's own key. Bind whenever the principal is a stored identity: a
session bound to a now-dead key is refused on its first request, which
is what the binding is for.
* s3: gate the vended-session key binding on the authenticated caller type
lookupByIdentityName(principal) named only that some identity exists, not
that the request signed with a stored key: an STS session or OIDC principal
that collides with an identity name got a session bound to its temporary
key, which vendedSessionKeyLive then refused on every request. Verifiers
now record the verified key as a stored credential, and the vendor binds
only when that marker is present — keeping the binding even when the key
was revoked between authentication and the vend.
* s3: enforce the vended key binding under a dedicated claim
Sessions minted before the binding existed carry the caller's key in the
audit-only SourceAccessKeyId — including temporary STS keys that can never
resolve in the identity table. Enforcing on that field would end live table
credentials at upgrade; the binding now travels under BoundAccessKeyId,
leaving legacy sessions unbound as before.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* s3: bind vended sessions to a credential fingerprint
The key id alone cannot identify a credential: an operator can delete a
key and re-create it under the same id with a different secret, reviving
sessions minted for the original. Verifiers now record a fingerprint of
the secret at authentication; the session dies unless it still matches.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* s3: keep recording the caller's key on every catalog session
SourceAccessKeyId was unconditional audit metadata before the binding
existed; gating it on the stored-credential marker dropped the record
for callers that authenticated another way. With the binding under its
own claim there is no ambiguity left, so the field records the caller's
key again regardless of how it authenticated.
* s3: key the vended-session credential fingerprint with the signing key
The bound-credential fingerprint travels inside the vended session
token. As a plain SHA-256 of the secret it gave any token holder an
offline guessing oracle for weak secrets, and CodeQL flags it twice.
Store the verified secret in the request marker instead and HMAC it
with the STS signing key at vend and at validation. The claim can no
longer be mined, and the per-request auth path stops hashing secrets.
---------
Co-authored-by: Javier de la Torre <jatorre@lakehousebox.com>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* s3lifecycle: schedule ExpirationDays from version-file writes
On a versioned bucket a PUT lands in the meta-log as the version file
plus the .versions/ directory update — there is no bare-key event, so
the version-folder skip dropped every version-file event for all kinds
but ExpiredDeleteMarker. ExpirationDays therefore only fired via the
recovery walker's cold-start / rule-change pass: once a shard had a
saved cursor, no daily run ever scheduled another delete marker.
A version-file write is the latest for its logical key at write time,
so route it through the same event-driven evaluation the regular path
runs. The match carries the file's identity with an empty VersionID —
the dispatcher resolves the current latest at due time, and a fresher
PUT meanwhile resolves to STALE_IDENTITY there.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* lifecycle test: versioned Expiration fires on the second daily run
Establishes the shard cursor with a first pass, then PUTs and backdates
a new versioned object and runs again. The delete marker must appear
without a rule change forcing a recovery walk.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* ci: run TestLifecycleVersionedBucketExpiresOnSecondRun in the lifecycle matrix
---------
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* volume: drop index entries that resolve to a different needle
An index row whose offset lands on a needle with a different id and a
different size cannot have been produced by any write path: every appender
indexes a key at the offset and size of the record it wrote, and
WriteNeedleBlob keeps its indexed size consistent with the blob header.
Such rows are corruption (for example an offset replayed with stale high
bytes) and until now made the volume reject every write for the key with
"mismatching cookie" while reads resolved to the wrong needle.
When the duplicate check finds the indexed record belongs to a different
key and size, drop the stale entry and let the append re-index the key.
Same-key conflicts still fail the cookie check, and entries staying
consistent with a re-keyed blob keep their existing semantics.
* volume: verify the indexed needle header before streaming reads
The buffered read path re-parses the record and rejects a needle whose
stored size disagrees with the index, but the streamed path serves data
bytes without ever looking at the header, so a stale index row would
stream another needle's payload. Probe the header once up front and
report a miss when the record is neither the requested key nor the
indexed size, the same staleness test the write path applies.
* rust volume: drop index entries that resolve to a different needle
Mirror of the Go volume change: when the duplicate check finds the
indexed record belongs to a different key and size, delete the stale
entry and let the append re-index the key.
* volume: overwrite a stale index entry on commit, not up front
Deleting the stale row during the duplicate check published a tombstone
before the replacement append and, for durable writes, before its fsync.
A failed append or sync then rolled back the data but left the key
deleted. The publish step already overwrites any older offset, so leave
the entry in place and let it be replaced once the record commits.
* volume: report streaming header-probe failures to the io error tracker
A failed header read returned before the tracked body-read loop, so
repeated EIO on the probe never reached checkReadWriteError. Skip io.EOF
the way the loop does: it is a truncated record, not a disk fault.
* rust volume: overwrite a stale index entry on commit, not up front
Mirror of the Go volume change: the staged delete published a tombstone
before the replacement append and its .dat flush, so a failed append or
sync left the key deleted. publish_write already overwrites any older
offset once the record commits.
* rust volume: report needle meta read failures to the io error tracker
read_meta is the header probe ahead of a streamed body: an EIO there
escaped check_read_write_error entirely, so a failing disk could not
trip the tolerance. Report the outcome the way read_full already does.
* volume: retry the wrapped offset before judging an index entry stale
A 4-byte index value can index past the 32 GiB wrap: the low offset may
hold another needle while the real record sits at +MaxPossibleVolumeSize,
which is where reads already retry on a size mismatch. Skipping the
cookie check on the low-offset record alone would accept a foreign-cookie
write over a live key. Retry the wrapped header read first, the same
condition the read path uses; a wrapped read that fails for reasons other
than EOF stops the write like the low-offset read does.
* volume: probe the streamed needle header under the access lock
With HasSlowRead the access lock is dropped after the index lookup, so a
compaction commit could swap the backend between the lookup and the
header probe and fail a valid download on the old offset. Take the read
lock for the probe and resolve the compaction revision first, the way
the streaming loop re-resolves it on every page.
* rust volume: retry the wrapped offset before judging an index entry stale
Mirror of the Go volume change: probe offset + MAX_POSSIBLE_VOLUME_SIZE
under the 4-byte-offset build before treating a mismatched record as
stale, so a live wrapped entry still gets its cookie checked.
* rust volume: do not clear the io error streak on a successful meta read
read_meta runs before the payload read in read_needle_for_get, so recording success there reset the streak before every failing download and the volume could never reach quarantine. Report failures only; read_full still records success after a complete read.
* volume: record a meta-only read's success against the io error streak
A meta read no longer clears the streak because a payload read still
follows on GETs, but requests the meta read ends outright — HEAD answers
and cookie mismatches — never reached the body path, so their success
left earlier failures counted. Note the success on those terminal paths;
mirrored in the Rust volume server.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* rust volume: report streamed reads to the io error streak
Streamed GETs never touched the tracker: a failing chunk read went
uncounted and a completed download never cleared the streak, unlike
readNeedleDataInto which does both.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* volume: reset the io error streak only at terminal meta reads
A streamed GET's metadata read runs before its payload read, so clearing
the streak there erased the count before every failing download and the
volume could never reach quarantine. Record the meta-read success only
where the request actually ends: must-meta-only reads, HEAD responses,
and cookie mismatches — matching the Rust volume server's placement.
* rust volume: record the streamed read's success at the final frame
Hyper can complete a Content-Length response without polling the body
again, so a success recorded only when the stream is next polled could
be skipped, leaving an earlier io error streak in place after a
successful download.
* volume: record a meta-only read's success on conditional 304 replies
* rust volume: record a meta-only read's success on conditional 304 replies
* volume: reset the io error streak only after the checksum passes
A streamed read that ends in a CRC mismatch still cleared the EIO
streak, so corruption could mask progress toward quarantine. The Rust
side records success only after the final frame's checksum verifies;
move the reset below the CRC check to match.
* volume: satisfy codespell in the meta-read comment
* rust volume: report range reads to the io error streak
* rust volume: cover the range read's io error reporting
---------
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* filer: do not stall startup waiting out the leader-settle window
ListExistingPeerUpdates re-polled the leader for up to 10s when it
reported zero filers. A fresh single-filer cluster can never gain peers,
so every filer start paid the full settle timeout before listening —
past the 6s health window in weed mini and short probe budgets.
Take the leader's first answer instead; the fallback fullest-answer
scan below still covers a leader that is mid-convergence.
* filer: bound leader-settle re-poll below the startup health window
* filer: update ListExistingPeerUpdates doc for empty-leader fallback
* filer: re-check empty membership at the announce gate before registering
A fresh filer that scans empty membership during a master restart or
re-election skipped bootstrap and announced immediately, serving missing
files until a peer happened to reconnect — and a late peer only got an
incremental subscription, so pre-existing metadata was never backfilled.
The announce gate now re-polls ListExistingPeerUpdates while the scan can
still be disproved (10s), backfills from any late peers before their
streams start, and then waits for their catch-up like startup peers. A
genuinely empty cluster pays a delayed master listing rather than a
stalled startup, since this runs after the ports already serve.
* filer: overlap the late-peer copy cutoff and stream start
A peer event stamped just behind this filer's clock could land after
the traversal read but carry a timestamp below the stream start —
the startup scan margins both cutoffs by LogFlushInterval for exactly
this skew; mirror it for peers discovered at the announce gate.
* filer: bound the late-peer bootstrap copy by the shared announce deadline
MaybeBootstrapFromOnePeer now takes the caller's context so a stalled
peer metadata copy cannot out-wait the announce cap, and the bootstrap
re-check shares one announceCatchUpCap deadline with the catch-up wait
instead of each spending a full minute.
* filer: merge bootstrap entries by commit time instead of blind insert
The late-peer copy runs while gRPC already serves, so a blind InsertEntry
could replace a write the local filer just accepted. Routing each entry
through applyTraversedEntry orders it against the local commit time under
the replay locks — a fresher local write survives the backfill.
---------
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* iam/oidc: reject reused jti claims with a bounded replay cache
* iam/sts: pass jti replay settings through convertToOIDCConfig
* iam/oidc: keep consumed jtis and stop the cleanup worker across re-init
Re-initializing a provider reset the JTI store, so an already-used token
could be presented again after a config refresh; the store is now
allocated once and survives Initialize. The cleanup worker is stopped on
every re-init (previously only when protection stayed enabled), is
started only after the rest of Initialize has succeeded, and Shutdown no
longer consumes a permanent sync.Once, so a re-initialized worker can be
stopped again.
* iam/oidc: enforce the jti replay cache bound for every token
Tokens without a usable jti claim skipped the cache entirely and stayed
reusable; they are now keyed by the token's own fingerprint so a repeat
presentation is still a replay. Tokens without exp previously failed as
expired on first use; their entry now gets a bounded retention. The
replay test-and-set now runs before any capacity work so replayed tokens
cannot each trigger a full-cache scan, the entry count is incremented as
part of the insert so concurrent validations cannot overshoot the bound,
and a guaranteed serialized sweep runs before a capacity rejection while
opportunistic sweeps are rate limited and confined to one at a time. Raw
token identifiers are no longer written to logs or errors, a stable
fingerprint is logged instead.
* iam/sts: shut down providers dropped by the managed-provider swap
Each managed OIDC provider runs a background JTI cleanup worker, but
SetIAMManagedOIDCProviders replaced the provider map without stopping
the old providers, so every refresh leaked goroutines. Providers absent
from the new map are now shut down; providers that stay keep running.
* iam/integration: reuse providers across refreshes for unchanged records
Every refresh rebuilt providers from their records, so a refreshed
provider forgot the JTIs it had already consumed and an already-used
token could be presented again. Records are now fingerprinted and an
unchanged record reuses its provider, keeping the consumed-token cache;
changed records are rebuilt, removed records are dropped, and the swap
in STS releases the workers of whatever is no longer referenced.
* iam/oidc: consume replay entries only on the single-use exchange path
Bearer authentication validates the same token on every request, so
consuming it in shared ValidateToken/Authenticate made every token
single-use and rejected all but the first request. The replay check now
runs only for the STS exchange via AuthenticateSingleUse; bearer paths
validate without consuming. A token without exp also no longer gets a
bounded replay entry that ages out while the token stays valid: its
entry is kept until the capacity bound evicts it, so it can never become
usable again.
* iam/oidc: keep the mock provider on the mock path for single-use calls
* iam/oidc: require exp on single-use tokens
A cache slot frees only by expiry, so an expiry-less entry would hold it
forever; a burst of distinct tokens could exhaust the bound permanently
and block every later exchange.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* s3: give each web-identity test token a unique jti
Subtests minted tokens with identical claims inside one second, so the
exchange saw the same token fingerprint twice and the second call read
as a replay. Each exchange now carries its own jti.
* iam/oidc: carry the consumed-token replay store across provider rebuilds
A record change rebuilds the managed provider with a fresh JTI store, so a token consumed before the rebuild could be presented again until it expired. The new provider adopts the replaced provider's replay store before it is published, and the replaced provider is shut down so only one cleanup worker sweeps the shared store.
* iam/oidc: stop the cleanup worker of providers dropped from the store
A refresh removed the provider object from the managed cache but left its JTI cleanup goroutine running for the life of the process.
* iam/oidc: merge a provider's consumed replay entries into another's store
MergeReplayStateFrom folds one provider's consumed-token entries into
another's, skipping already-expired ones. Unlike AdoptReplayStateFrom it
swaps no pointers, so it is safe on a provider still serving validators.
* iam/integration: share one provider and replay store across identical issuer records
Records for the same issuer URL with identical validation settings now
reuse a single provider, so a token consumed through one account's
record is rejected through the other's — previously each record owned a
separate jti store and a spent token could be replayed cross-account.
A superseded provider's consumed entries merge into the shared store so
no rejection is lost, and a removed or rebuilt provider is only shut
down once no remaining record references it.
* iam/oidc: hand the superseded provider's live replay store to the shared one
* iam/oidc: carry merged replay entries without charging the capacity bound
* iam/oidc: key the jti-less replay fingerprint on the signed input
Hashing the whole token lets ECDSA signature malleability (s -> N-s)
mint a second cache key for identical claims, so a consumed token could
be replayed for another exchange. Key the fallback on header.payload
instead.
* iam/oidc: publish the jti cache bound before the enabled flag
Initialize stored jtiEnabled ahead of jtiMaxSize, so a provider flipped
from disabled to enabled exposed enabled=true with a zero bound for a
window; a single-use validation landing there would compute
newCount > maxSize and reject with "JTI cache capacity exceeded". Store
the bound first so the flag is the last thing published.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* s3: bound ListMultipartUploads memory to the page size
listMultipartUploads loaded every open upload in the bucket into a slice
and selectMultipartUploads filtered and sorted a second full copy, so one
request cost O(total open uploads) heap regardless of prefix, key-marker,
or max-uploads; concurrent calls multiplied it and could OOM the gateway.
The response is ordered by the object key kept in extended attributes, not
the upload id the filer sorts by, so every entry still has to be seen.
Streaming the listing into a bounded top-K collector keeps at most
maxUploads (capped at maxUploadsList) candidates, bounding heap by the
page instead of the folder. A caller-supplied max-uploads above the
server cap now also paginates rather than accumulating unboundedly.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* s3: cover bounded ListMultipartUploads accumulation
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* s3: report the effective max-uploads cap in ListMultipartUploads
* s3: cover ListMultipartUploads retry through the collector
---------
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* worker/ec: retry the pinned volume copy after a mid-copy compaction
The EC task pins .idx/.dat copies to the CompactionRevision read from
ReadVolumeFileStatus so the pair comes from one volume generation. When
vacuum compaction lands between that read and either CopyFile, the
server rejects the copy with "volume N is compacted" and the whole task
fails — on clusters with queued maintenance this makes most EC tasks
die to a race they cannot avoid. On a compaction mismatch the copy now
restarts under a freshly read revision, up to a bounded number of
attempts; other errors still fail immediately.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* worker/ec: read the retried copy's expiry from its own file status
A mid-copy compaction makes the source reload the volume, and the fresh
ReadVolumeFileStatus can carry a different expireAtSec than the
pre-compaction read. Keeping the attempt-0 expiry could let the EC
volume outlive its source TTL.
* worker/ec: cover the compaction retry with a stub volume server
* worker/ec: also observe revised .idx bytes in the compaction retry test
* worker/ec: read the fake source under its mutex in the retry test
---------
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* operation: renew the cleanup JWT when a fan-out rollback delete is rejected
A stalled holder can out-wait the assignment JWT's 10s expiry, so the
type=replicate delete that rolls back landed copies comes back 401 and the
needle leaks. Retry the delete once with a fresh token minted through the new
LookupJwtFunc hook when a holder answers unauthorized.
* s3api: wire a cleanup-JWT renewer into chunked uploads
Keep the master client on the server and use it to mint fresh volume JWTs for
rollback deletes that run past the assignment token's expiry.
* admin: wire a cleanup-JWT renewer into file-browser uploads
* operation: bound the cleanup-JWT renewal with a per-attempt deadline
A rollback delete that hits 401 renewed the token through a background
context, so a master outage could park the wait for a connection and the
lookup RPC inside the failed upload's cleanup path. Thread a context
through LookupJwtFunc and LookupJwt, and bound each renewal with a 10s
deadline covering both the master wait and the lookup call.
* operation: let every holder's 401 retry with a renewed cleanup JWT
A single renewed flag meant only the first rejecting holder refreshed the
token; a holder that stalled past the renewed JWT's lifetime left later
copies stranded with a 401 they could not resolve. The retry limit is now
per holder — each delete is attempted once with the current token and once
more after a bounded renewal — so a slow middle holder cannot strand the
rest.
* operation: log the volume server's error body on a failed cleanup delete
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* operation: count test upload attempts atomically
The chunk upload loop fans AssignFunc out to parallel goroutines, so the
plain int counter in the partial-results test raced under -race.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test/kafka: bump klauspost/compress to v1.20.3 in simple-publisher
Patch an s2 integer-overflow / out-of-bounds write affecting
klauspost/compress <= 1.18.6; matches the version the main module
already uses. The go directive rises to 1.25, still under the ^1.26
toolchain the kafka test workflows run with.
* seaweed-volume: bump aws-smithy-json to 0.62.7
Patch an uncontrolled-recursion denial of service in the unknown-key
skip path of aws-smithy-json <= 0.62.6 (reachable over the S3 SDK's
response parsing). The smithy runtime family moves together:
aws-smithy-runtime-api 1.12.3 and aws-smithy-types 1.5.0, all selected
within the workspace's Rust 1.91.1 compatibility bound.
* fuse integration: add POSIX compliance test suite
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* ci: add posix-compliance workflow
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fuse integration: skip non-executable weed path candidates
* fuse integration: size readv buffer to the written segment
* fuse integration: reference posix_Makefile explicitly in recursive make calls
GNU make does not propagate -f through MAKEFLAGS, so bare $(MAKE) re-read the default Makefile in this directory.
* fuse integration: fail posix test targets when go test fails
Every suite pipes go test through tee under /bin/sh, so the recipe status was tee's and test failures stayed green. Run recipes under bash -o pipefail and emit the color escapes with printf since bash echo does not expand them.
* fuse integration: run only benchmarks in benchmark-posix
go test -bench=. without -run also executes the whole suite, mixing compliance output into the benchmark log.
* fuse integration: emit single-number test counts in reports
grep -c prints 0 and exits nonzero when nothing matches, so '|| echo 0' produced two lines that broke the expr arithmetic and JSON fields. awk prints exactly one count either way.
* fuse integration: skip the sendfile test on macOS
macOS sendfile(2) only accepts a socket destination; the test copies into a regular file and always gets ENOTSOCK.
* fuse integration: honor TestConfig and own only self-started resources
NewFuseTestFramework dropped its config argument and Cleanup read a fresh
DefaultTestConfig, so SkipCleanup never did anything. Cleanup also skipped
teardown whenever TEST_SKIP_CLUSTER_SETUP was set, even when TEST_MOUNT_POINT
was empty and Setup had started the cluster itself. The framework now keeps
the config it was built with, runs external-mount tests inside a per-run work
dir under the mount, and only skips teardown when an external mount was
actually used.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fuse integration: pass the debug flag before the subcommand
-v is a weed global flag; appended after master/volume/filer/mount it lands
on the subcommand flagset, which rejects it and the process dies at flag
parsing whenever EnableDebug is on.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fuse integration: require a real mount before running tests
Setup creates the mount point before weed mount starts, so statting and
listing it succeeded on the plain directory and tests ran against local
disk whenever the mount was slow or died. waitForMount now waits until the
mount point's device id differs from its parent's.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fuse integration: resolve the weed binary to an absolute path
The started processes run with cmd.Dir at the framework's temp dir, so a
relative candidate like ../weed no longer resolves. Prefer PATH, then probe
the usual build outputs and return absolute paths only.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fuse integration: build the external test work dir with MkdirTemp
The suite staged cloned tools and generated scripts under a predictable
/tmp path another user could pre-create and swap mid-run.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fuse integration: use the pjdfstest tree setup-pjdfstest installs
The test gated on a pjdfstest binary in PATH but then proved a suite it
cloned itself, so a tree installed by make setup-pjdfstest was never found.
Look for the built suite under external_tools first, then clone into the
work dir as before.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* ci: require a real mount before posix tests
The wait loop fell back to listing the mount point, which passes on the
directory created before weed mount runs — a dead or slow mount meant the
suite exercised local disk. Only mountpoint counts now, and a dead mount
process fails fast instead of timing out.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* ci: collect failure logs before cleanup deletes them
The always-on cleanup step removed /tmp/seaweedfs before the failure-only
debug step could copy the service logs, so failed runs uploaded nothing.
Run cleanup after log collection.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fuse integration: point MOUNT_POINT at TEST_MOUNT_POINT
Targets that work against MOUNT_POINT (pjdfstest, nfstest, fio configs)
ignored TEST_MOUNT_POINT and ran against the local-disk default while the
go tests used the external mount.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fuse integration: skip the permission-denied check as root
Root bypasses mode-bit checks, so writing the read-only file succeeded and
the assertion failed in root-run environments like containers.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fuse integration: align the posix docs with the suite
The doc listed a docker-test-posix target and make variables that do not
exist, and ran go test against single files that cannot compile alone. The
roadmap also still claimed the platform-specific tests this suite now
implements were all skipped.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fuse integration: honor SkipCleanup in the mini-based framework too
Same gap as the fuse-package framework: Cleanup read a fresh
DefaultTestConfig, so a test's SkipCleanup setting was ignored.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fuse integration: normalize config in Setup and clean the work dir too
Setup(nil) panicked on config.EnableDebug and external-mount Cleanup left
the temp dir behind; use the stored config for the start* calls and remove
tempDir alongside the external work dir.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* ci: point external posix suites at the workflow FUSE mount
* fuse integration: create the fio target directory before running
* fuse integration: use unix.O_DIRECT instead of a hard-coded x86 value
0x4000 is O_DIRECT only on x86 and amd64; on arm64 it is O_DIRECTORY.
The test module already depends on golang.org/x/sys.
* fuse integration: run autoreconf and configure before building pjdfstest
A fresh pjdfstest clone is autotools-based and has no Makefile, so the
Go-side setup fallback always failed and the suite silently skipped.
* ci: require external posix tools only when setup installed them
setup-external-tools always exits 0 because each sub-target tolerates
failure, so POSIX_REQUIRE_EXTERNAL_TOOLS was set even with no tools
installed and the suite then hard-failed on the missing suites.
* ci: fail the posix workflow when the mount point is not a FUSE mount
The file write/read check passes on the plain directory created before
weed mount, so a dead mount let the whole suite grade the local disk.
* fuse integration: skip when the host has no FUSE device node
Setup failed the test outright on machines without /dev/fuse instead of
skipping; cover every suite entry point by skipping inside Setup.
* fuse integration: drop the chmod 666 /dev/fuse troubleshooting tip
World-writable device access is a security risk; the fuse group
membership on the line below is the safe fix.
* fuse integration: require go 1.21+ in check-go, not 1.20+
* fuse integration: refuse external suites on an unmounted path
* fuse integration: verify TEST_MOUNT_POINT is a real mount
* fuse integration: clean TEST_MOUNT_POINT before the parent-device check
filepath.Dir of a trailing-slash path returns the mount itself, so a mounted TEST_MOUNT_POINT ending in / was refused.
* fuse integration: detect WinFsp mounts on Windows in mountedAt
A directory mount is a reparse point and a drive mount is a volume root; a plain directory is now refused instead of accepted.
* fuse integration: strip a trailing slash off MOUNT_POINT in require-mounted
The mount-table grep fallback on hosts without mountpoint(8) would not match a slash-suffixed entry.
* fuse integration: require an external mount to be a FUSE filesystem
A device-ID boundary alone does not prove FUSE: a tmpfs mount would pass mountedAt and report its results as FUSE compliance. Check the filesystem type (statfs magic on linux, fstypename on the BSDs); platforms without a check keep the mount and write verification.
---------
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* volume: never compact a volume tiered to remote storage
A tiered volume's data backend is the remote object, but compaction reads
the local .dat path. When tiering removed the local .dat, every vacuum of a
tiered volume with garbage fails with "no such file or directory", and the
master's sweep and the worker's vacuum task keep selecting it. When the
local .dat was kept, the compaction succeeds against the stale copy and the
commit reloads the volume to read the remote object (old layout) through the
compacted index, so live needles stop reading.
Mirror the Rust volume server (#11545):
- CompactByIndex and CompactByVolumeData refuse a volume with a remote file
before creating .cpd/.cpx ("volume N is tiered to remote storage, cannot
compact").
- CommitCompact refuses under the data-file locks when the volume became
tiered during the copy, and removes .cpd/.cpx.
- Store.CheckCompactVolume reports no garbage for a tiered volume, so the
master's vacuum and volume.vacuum do not select it.
- The worker's vacuum detection skips a volume with a remote copy, as balance
and EC detection already do.
Tests (each fails on master): TestCompactRefusesTieredVolume,
TestCommitCompactRefusesVolumeTieredDuringCopy (on master the commit
succeeds and a surviving needle no longer reads),
TestPickPrimaryReplicaSkipsTieredVolumes.
* volume: skip tiered replicas in vacuum selection, keep their real ratio
Reporting a zero garbage ratio for a tiered volume hid it from
volume.tier.compact, whose preflight reads that ratio while the volume is
still remote, and did not exclude it from a vacuum with a zero threshold.
Report the real ratio again, as the Rust volume server does, and have the
master's vacuum check skip a replica whose volume info names remote
storage, whatever the threshold or request.
When CommitCompact refuses a volume tiered during the copy, also close the
temporary needle map and remove .cpldb: nothing else would, since no later
vacuum selects the volume.
The tiered-compaction test no longer depends on random needle sizes
reaching a 0.3 garbage ratio.
* volume tier: do not attach a copy of a .dat a compaction replaced
The upload copies the .dat, then records the remote file in the .vif and
switches the backend. A compaction commit in between swaps in a compacted
.dat and index; the upload then served the old-layout remote object through
the new index, so live needles stopped reading. CommitCompact's tiered
check cannot see this: the volume is not tiered until the switch.
AttachRemoteDatFile records the remote file and switches the backend under
the data-file locks, and refuses when the backend is no longer the file
that was copied. The upload then removes the unused remote copy.
* volume tier: keep a committed .vif entry when only the dir fsync fails
AttachRemoteDatFile rolled back the in-memory remote entry on any
SaveVolumeInfo error. A NotCrashDurableError comes after the rename, so
the .vif on disk already names the copy; with the entry gone, the upload
handler deleted that object, and a reload would serve from a missing
remote file. Keep the entry and switch to the copy as the .vif says,
and still return the error so the caller keeps the local .dat.
* vacuum worker: filter tiered replicas instead of skipping the volume
A partially tiered volume is the leftover of an interrupted
volume.tier.upload; skipping the whole volume left garbage on its local
replicas unreclaimed. Per-replica filtering matches the master's
batchVacuumVolumeCheck and is safe: a tiered replica refuses both
compaction and commit.
* filer: sync leveldb entry writes when the path rule asks for fsync
The fsync storage rule only reached the volume upload; the metadata
writes behind an acknowledged create, update or delete still went to
the leveldb journal with Sync:false, so a power loss could roll back
an acknowledged bucket or object entry.
A write-side flag on the entry context carries the request: the filer
sets it from the path's storage rule in CreateEntry, UpdateEntry,
EnsureDirectoryEntry and DeleteEntryMetaAndData, and the HTTP write
path sets it for an explicit ?fsync=true. The leveldb stores write
with Sync:true when the flag is present.
* volume: flush the .idx before acknowledging a durable write
A write with fsync synced only the .dat: the index row written on
publish stayed in the page cache, so a power loss could leave the
needle's bytes durable with nothing indexing them.
The durable paths now run one .idx sync after publish, before the
requests complete — one per syncWrite and one per batch run, paired
with the .dat sync, the same split the Rust volume server makes
between flush_dat and flush_idx. A failed .idx sync cannot be rolled
back, so it stops the volume taking writes and fails the requests
instead of acknowledging an index entry that may not survive.
---------
Co-authored-by: Javier de la Torre <jatorre@lakehousebox.com>
* s3api: support s3:RequestObjectTag/<key> in bucket policies
The tags come from X-Amz-Tagging only where the handler stores them from
there: not on PutObjectTagging, which reads the body, and not on a copy
that keeps the source's tags. For a repeated key the last value counts,
since that is what CopyObject stores.
* s3api: extract RequestObjectTag only where the handler stores it
The header was honored on any request carrying it, so a tag attached to
a GET, DELETE, part upload, multipart completion, directory marker or a
PUT subresource satisfied a tag-conditioned policy the object never
receives. Extraction now follows the handlers: PutObject, a copy under
X-Amz-Tagging-Directive: REPLACE, and CreateMultipartUpload store the
header; a plain object PUT is identified by carrying only X-Amz-*
signing parameters.
* s3api: exclude multipart part PUTs before the copy-source check
A PUT carrying X-Amz-Copy-Source plus a numeric partNumber and uploadId
routes to CopyObjectPartHandler, which stores a part and never reads
X-Amz-Tagging. Checking the copy source first let such a request satisfy
an s3:RequestObjectTag condition under REPLACE with tags that are never
stored. Mirror the router: the part handlers match only when both
partNumber (digits only) and uploadId are present, so a lone or
non-numeric partNumber still falls through to the storing handlers.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* s3api: gate tag extraction on the routed subresource, not query key shape
Rejecting every PUT whose query held a key outside X-Amz- dropped
X-Amz-Tagging from ordinary object PUTs that carry arbitrary parameters —
Signature V2 presigns (AWSAccessKeyId/Signature/Expires) among them — so a
tag-conditioned Deny could be bypassed by appending an unrelated query
parameter while putToFiler still stored the tags. Exclude the object PUT
subresources the router actually dispatches (tagging, renameObject, acl,
retention, legal-hold) ahead of the copy-source check instead.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* s3api: cover the routed-subresource and part-copy tag extraction cases
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Chris Lu <chris.lu@gmail.com>
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* s3: reject requests carrying more than one routed subresource
Routing picks the first matching subresource route while authorization
resolves a single action from the query, and the two can disagree. A
bucket-level PUT ?tagging&policy is authorized as s3:PutBucketTagging but
served by PutBucketPolicyHandler, letting a tagger rewrite the bucket
policy; the same shape lets a list grant read the policy via
?list-type=2&policy or ?versions&policy, and lets an object ACL writer
retag via PUT ?acl&tagging. Declining ambiguous requests keeps the
authorized action and the served handler identical.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* s3: reuse hasAmbiguousSubresource in the routed-subresource guard
The standalone key list dropped versioning, so ?versioning&policy still
reached route matching and auth with two operations. The bucket-level
check already keys off operationSubresources plus bucketQueryActions;
running the same predicate keeps the lists from drifting apart.
---------
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Forwarding a lock to its owner filer is routine in a multi-filer setup,
so the forward and acquire messages move from V(0) to V(3). The acquire
message is only logged when the forwarded response holds a renew token
and no application error. Transport failures still log at V(0).
Co-authored-by: zouyixiong <3390062+zouyixiong@users.noreply.github.com>
WriteResponse wrote the payload even for HEAD requests and statuses
that disallow a body (1xx, 204, 304), which made net/http return
ErrBodyNotAllowed and logged 'write err: http: request method or
response status code does not allow body' for ordinary requests.
Generated with [Devin](https://devin.ai)
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* filer leveldb2: read cache and buffer sizes from filer.toml
The leveldb2 store hardcoded its block cache (32MiB) and write buffer
(16MiB) and offered no way to tune the table block size or compaction
table size. Optional block_cache_size_mb, write_buffer_size_mb,
block_size_kb and compaction_table_size_mb keys under [leveldb2] now
override them; zero or negative values keep the previous defaults.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* filer leveldb2: document tuning options in scaffold filer.toml
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* filer leveldb2: saturate size options instead of overflowing int
The option fields take int, so a MiB config value past math.MaxInt/MiB
(e.g. 2048 on 32-bit) wrapped negative and goleveldb would panic sizing
the memtable. Clamp the product instead of the input.
* filer leveldb2: reject size options too large for int bytes
Clamping a MiB setting at math.MaxInt silently hands goleveldb a
different size than configured; fail Initialize on values that cannot
fit instead.
* filer leveldb3: same tunable cache, buffer, block and table sizes
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* filer leveldb2/3: exercise oversized-option rejection on 64-bit too
The tests skipped whenever int was 64-bit, but write_buffer_size_mb
values above MaxInt/1MiB still overflow there, so the rejection path
had no coverage on the primary platform. Use MaxInt/1MiB + 1 so the
rejection triggers on every platform.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* volume.fsck: delete missing filer entries through a worker pool
-reallyDeleteFilerEntries issued one synchronous HTTP DELETE per orphan
path inside the BFS consumer, so a million-entry purge ran one request
at a time. Queue the confirmed paths onto a bounded pool (default 16,
-tunable via -purgeConcurrency) so deletes run in parallel while the
traversal continues.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* volume.fsck: serialize command output across purge workers
* volume.fsck: serialize purges that share a hard link id
* volume.fsck: cover the purge worker pool
* volume.fsck: take the purge shard remainder in unsigned arithmetic
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
The filer only listens on its unix socket when GOOS is not windows, so
with a filer socket path configured on Windows the S3 server dialed a
socket that never exists.
Fixes#7278
* s3, iceberg: record the access key a request was signed with
Several access keys can belong to one identity, but nothing downstream of
authentication says which one signed a request: the audit log carries the
identity name only, and every table credential the Iceberg catalog vends for
that identity gets the same session name. When an identity is shared (one key
per person or per service), the log cannot tell the holders apart.
The verifier that matched a credential now records that credential's access
key id in the request's identity holder (verifyV4Signature for header,
presigned, streaming and STS requests; both SigV2 verifiers). Nothing re-reads
the request for a key, so a second, unverified key elsewhere in the request
(a junk X-Amz-Credential on a SigV2 request, a junk SigV4 header on a SigV2
presigned URL) is never recorded.
- s3err.AccessLog gains access_key_id (omitempty): a static key's id, or an
STS session's temporary key id. Every other field is unchanged.
- The Iceberg and Lance catalogs install the holder before authenticating a
SigV4 request; a Bearer token records the key it was minted from (the token
is verified with a key derived from it); a Lance x-api-key records its key.
- VendTableCredentials ends the session name with "-k" and the first 12 hex
digits of SHA-256 of the key id (the table part gives way within the
64-character limit; the tag is never cut), and carries the key id in the
session token (src_ak, SessionInfo.SourceAccessKeyId).
Tests: auth_access_key_attribution_test.go and auth_access_key_shapes_test.go
(two keys on one identity, the deny path, SigV2 and presigned SigV2 with a
junk key elsewhere, streaming seed, STS session, failed signatures record
nothing), s3api_iceberg_session_name_test.go, iceberg and lance Auth tests,
sts source_access_key_test.go.
* s3: record the access key of a verified POST-policy form and of a retargeted signature
Two verified signatures left access_key_id empty in the audit log:
- A browser POST upload is verified in PostPolicyBucketHandler from its form
fields by doesPolicySignatureV2Match / doesPolicySignatureV4Match, which did
not record the key. Both now take the request context and record the
credential's key once its signature verifies; a form that fails
verification records nothing.
- verifyV4Signature refuses a signature that verified for one virtual-hosted
bucket but was routed to another. The key was recorded only after that
check, so the refusal carried none. It is now recorded as soon as a host
candidate verifies.
Test: TestPostPolicyUploadRecordsTheSigningKey (SigV4 and SigV2 forms, and a
form with a wrong signature) and TestSignedVirtualHostRetargetRecordsTheSigningKey;
both fail before this change.
* s3: test VendTableCredentials end to end for the caller's access key
TestVendTableCredentialsRecordsTheCallersKey mints sessions through
VendTableCredentials with a configured STS service, the key recorded the way
the catalog's authentication records it, then validates each session token:
SourceAccessKeyId and the session name's key tag match the key, and a caller
without a key gets an untagged session. It fails if the vendor stops passing
either field.
---------
Co-authored-by: Javier de la Torre <jatorre@lakehousebox.com>
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* shell: volume.fsck reads the filer from one store snapshot when it can
volume.fsck collects the filer's chunk references by walking directory
by directory, so each directory is seen at a different time. A file
renamed from a directory not yet listed into one already listed is never
seen, its chunks look like orphans, and -reallyDeleteFromVolume deletes
live data. Checkpoint rotation and write-to-tmp-then-rename hit this.
Add an optional filer store capability, EntryScanner, that visits every
entry below a directory from one consistent snapshot. The SQL stores
implement it with one REPEATABLE READ read-only transaction read in
primary-key pages, so a concurrent rename is seen under exactly one of
its names. Key-value rows are skipped by their dirhash, a bucket
directory whose table is missing fails the scan (from the snapshot it
cannot be told from one dropped a moment ago), and hard links are
resolved in the same transaction. SQLite offers it only in WAL mode, on
its own connection.
TraverseBfsMetadata serves the snapshot when asked via request metadata
sw-traverse-mode=snapshot and reports the mode in a response header;
without the metadata it walks as before, so peer bootstrap and admin
export are unchanged. Snapshot responses carry no entry_ts_ns, whose
per-entry lookup would cost more than the scan.
volume.fsck requests the snapshot and falls back to the walk on an older
filer or a store that cannot scan.
* filer, shell: fail a scan that cannot start, not fall back to the walk
A transient BeginTx failure and a preliminary GetEntry error were treated
like a missing capability or missing root, so fsck would take the
directory walk - the very path that misses renamed-away entries. Only a
genuinely unsupported scan or a confirmed-absent root may fall back.
* filer: only snapshot-listed bucket names keep a missing table fatal
A cached-only name, whose bucket may have been deleted between copying
the cache and opening the snapshot, is skipped when its table is gone.
* filer: roll back to a savepoint before skipping a cached-only table
On PostgreSQL a failed statement aborts the snapshot transaction, so
ignoring a missing-table error without the savepoint still left every
later cached-only table unreadable.
---------
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
* remote_storage/s3: use the AWS default credential chain when a source is configured
Type s3 forced anonymous credentials whenever no access/secret key was
set, so IRSA, container credentials, profiles and env keys were never
consulted. Keep anonymous (public bucket) access only when the environment
names no AWS credential source.
* remote_storage/s3: also recognize shared-config and default-profile env sources
* remote_storage/s3: resolve env credentials on their own transport
A guarded HTTP client is for the S3 endpoint; the remote credential
providers it would also serve reach link-local addresses that guard must
not dial. Profiles load shared config, and a test resolves env keys.
* remote_storage/s3: clear every detected credential source in the test
* remote_storage/s3: resolve the full default chain on the guarded client
The eager credential session used NewSession, whose chain covers only
env, shared-credentials and ECS/EC2 providers. Web identity and
assume-role profiles resolve through shared config, so IRSA never
worked when a custom HTTP client was pinned. Enable shared config and
carry the configured region so STS can sign assume-role requests.
---------
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* volume: report the needle read error with a failed read
A read that fails on the volume server, for example on a CRC mismatch,
returned a bare 500, so mount and filer logs only showed the status.
The volume server now writes the error into the response body, and the
read helpers add it to the error they return.
Co-authored-by: Bruce <82158894+half-life666@users.noreply.github.com>
* volume: keep the 500 bare on HEAD requests
writeJsonError's body write fails with ErrBodyNotAllowed under net/http
when the request method is HEAD, adding an error-writing log line on top
of the failed read. Send only the status, matching the no-error path.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* util/http: decode gzip-encoded error bodies in statusError
Full-chunk reads set Accept-Encoding: gzip themselves, so the
transport does not transparently decompress; a gzip-encoded error
body reached statusError as compressed bytes and produced an
unreadable message. statusError now takes the response and decodes
Content-Encoding before extracting the bounded message.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* volume: log the read error when the 500 stays bare
On HEAD requests the body write is skipped, which also skipped the
V(0) response log inside writeJson — a failed HEAD read left no
cause in normal volume-server logs. Log the error in the same
format before the bare 500.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* util/http: check the status before decoding gzip in GetAuthenticated
The gzip reader eagerly consumes the header when it is created, so
statusError saw a drained or corrupt body on a gzipped error response
and the server message was lost. Match the other read helpers and check
the status first.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* util/http: export StatusError
Other volume read paths outside this file need the same error body.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* volume: redact jwt query param when logging failed-read URLs
The new read-failure logs print the request URL, and volume reads accept
?jwt= as a credential, so a failed authenticated read would write the
bearer token to the volume server log.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* util/http: surface the error body on the remaining chunk-read paths
* volume: redact duplicate jwt params in failed-read logs
* operation: pass the lookup jwt on the chunk needle read
---------
Co-authored-by: Bruce <82158894+half-life666@users.noreply.github.com>
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
* volume.configure: keep the superblock in sync with the .vif
ConfigureVolume only rewrote the .vif's replication field, leaving the
.dat superblock stale. A mounted volume also kept reporting the old
placement in heartbeats, and its next compaction would rewrite the stale
placement into a fresh .dat.
Parse the placement once, then update the live volume when mounted: the
in-memory .vif copy and superblock, and the .dat header through the
volume's own backend under the data-file locks. For unmounted volumes
rewrite the header directly; remote-only volumes skip it since the .vif
carries the placement.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* volume.configure: update the .dat superblock in the Rust volume server
Mirror the Go ConfigureVolume change: after rewriting the .vif, rewrite
the replica placement in the .dat header too, so a remounted volume's
superblock, heartbeats, and later compactions agree with the .vif. A
missing .dat (remote-only volume) is skipped since the .vif carries the
placement.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* shell: bound the initial master connection wait behind -connectTimeout
weed shell waited on WaitUntilConnected with a background context, so
an unreachable master hung the command forever. The wait now runs on
a 30s (configurable) timeout and reports the failure instead.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* shell: stop the master reconnect loop when RunShell returns
KeepConnectedToMaster ran on context.Background(), so once RunShell
returned — including the new connect-timeout early return — the
reconnect goroutine kept dialing forever, and repeated RunShell calls
accumulated one each. Give it its own cancellable context, separate
from connectCtx whose deadline must not cut reconnects under a live
shell.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* shell: bound filer discovery by -connectTimeout too
The discovery WithClient and ListClusterNodes ran on the unbounded
background context, so a master disconnect after the initial connect —
or a persistently failing ListClusterNodes — could block startup past
the deadline. Reusing connectCtx bounds the whole startup phase.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* shell: report filer discovery failures instead of silently losing the filer target
* shell: give filer discovery its own budget after master connects
Discovery shared connectCtx, so a connection that landed near the
-connectTimeout deadline left ListClusterNodes almost no time and no room
for WithClient to retry. A fresh timeout keeps discovery bounded without
starving it.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* shell: cover the -connectTimeout master wait with a test
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* shell: note filer discovery in the -connectTimeout help text
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* volume: never compact a volume tiered to remote storage
A tiered volume's data backend is the remote object, but compaction reads
the local .dat path. When tiering removed the local .dat, every vacuum of a
tiered volume with garbage fails with "no such file or directory", and the
master's sweep and the worker's vacuum task keep selecting it. When the
local .dat was kept, the compaction succeeds against the stale copy and the
commit reloads the volume to read the remote object (old layout) through the
compacted index, so live needles stop reading.
Mirror the Rust volume server (#11545):
- CompactByIndex and CompactByVolumeData refuse a volume with a remote file
before creating .cpd/.cpx ("volume N is tiered to remote storage, cannot
compact").
- CommitCompact refuses under the data-file locks when the volume became
tiered during the copy, and removes .cpd/.cpx.
- Store.CheckCompactVolume reports no garbage for a tiered volume, so the
master's vacuum and volume.vacuum do not select it.
- The worker's vacuum detection skips a volume with a remote copy, as balance
and EC detection already do.
Tests (each fails on master): TestCompactRefusesTieredVolume,
TestCommitCompactRefusesVolumeTieredDuringCopy (on master the commit
succeeds and a surviving needle no longer reads),
TestPickPrimaryReplicaSkipsTieredVolumes.
* volume: skip tiered replicas in vacuum selection, keep their real ratio
Reporting a zero garbage ratio for a tiered volume hid it from
volume.tier.compact, whose preflight reads that ratio while the volume is
still remote, and did not exclude it from a vacuum with a zero threshold.
Report the real ratio again, as the Rust volume server does, and have the
master's vacuum check skip a replica whose volume info names remote
storage, whatever the threshold or request.
When CommitCompact refuses a volume tiered during the copy, also close the
temporary needle map and remove .cpldb: nothing else would, since no later
vacuum selects the volume.
The tiered-compaction test no longer depends on random needle sizes
reaching a 0.3 garbage ratio.
* volume tier: do not attach a copy of a .dat a compaction replaced
The upload copies the .dat, then records the remote file in the .vif and
switches the backend. A compaction commit in between swaps in a compacted
.dat and index; the upload then served the old-layout remote object through
the new index, so live needles stopped reading. CommitCompact's tiered
check cannot see this: the volume is not tiered until the switch.
AttachRemoteDatFile records the remote file and switches the backend under
the data-file locks, and refuses when the backend is no longer the file
that was copied. The upload then removes the unused remote copy.
* volume tier: keep a committed .vif entry when only the dir fsync fails
AttachRemoteDatFile rolled back the in-memory remote entry on any
SaveVolumeInfo error. A NotCrashDurableError comes after the rename, so
the .vif on disk already names the copy; with the entry gone, the upload
handler deleted that object, and a reload would serve from a missing
remote file. Keep the entry and switch to the copy as the .vif says,
and still return the error so the caller keeps the local .dat.
* vacuum worker: filter tiered replicas instead of skipping the volume
A partially tiered volume is the leftover of an interrupted
volume.tier.upload; skipping the whole volume left garbage on its local
replicas unreclaimed. Per-replica filtering matches the master's
batchVacuumVolumeCheck and is safe: a tiered replica refuses both
compaction and commit.
---------
Co-authored-by: Javier de la Torre <jatorre@lakehousebox.com>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
* upload: upload -dir files concurrently behind a new -c flag
Each file under -dir was submitted serially inside filepath.Walk, so
every small file paid a sequential master-assign plus upload round
trip. A bounded pool now runs those submissions concurrently while
keeping results in walk order; failures are reported per file and the
command exits non-zero when any upload failed.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* submit: close opened readers when a batch assign fails
SubmitFiles returns early when the assign RPC fails, before
FilePart.Upload gets a chance to close each opened file. Close the
readers on that path so repeated assign failures do not leak handles.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* upload: stream -dir results as each file completes
Batching the output until every upload finished meant an interrupted
run lost the file ids of completed uploads. A per-index ready channel
lets a printer emit each result in walk order as soon as it lands.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* submit: close the opened file when Stat fails
newFilePart returned the still-open handle with the error, and the
concurrent upload path records the error without ever reaching
SubmitFiles, leaking one descriptor per failing file.
* upload: still upload files collected before a walk error
A walk that aborts partway used to leave the already-uploaded files
behind; collecting paths first must not drop them, so upload what was
gathered and still exit non-zero.
* upload: submit batched files concurrently
SubmitFiles fetched all file ids in one assign RPC but then uploaded
the files one after another, so a multi-file list still spent its time
in serial per-file round trips even with -dir concurrency. Uploads now
run through a small worker pool while results keep their input order.
Verified end to end: a 5-file list uploads with consecutive file keys
and each stored needle matches its source bytes.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* upload: print per-file results before exiting on a partial batch failure
---------
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* storage: fail on short super block extra read even without an error
Some backends (e.g. s3_backend) clear io.EOF after a short read, returning
n < len(p) with a nil error. Checking n only when the error is non-nil
lets truncated extra data reach proto.Unmarshal with a zero-filled tail.
* storage: apply the same short-read check to the super block header
A backend that returns n < SuperBlockSize with a nil error previously
slipped through and parsed a partially zero-filled header.
* storage: test super block extra reads
Covers a round trip through ReadSuperBlock, a truncated extra region, a
backend returning io.EOF alongside a full read, and a backend returning
a short read with a nil error.
---------
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
* iam: optionally require an iam: grant for self-service access-key actions
AuthorizeIamAction lets every authenticated identity run CreateAccessKey,
DeleteAccessKey, UpdateAccessKey, ListAccessKeys and GetUser on itself with
no grant. On AWS these actions need an IAM policy (the usual one grants them
on arn:aws:iam::*:user/${aws:username}); nothing is allowed implicitly.
Where identities hold data-plane rights only, the implicit carve-out lets a
leaked key mint further keys for its own identity, with no cap, that keep
working after the operator deletes the key that leaked; it can also
deactivate or delete the keys the operator issued.
A new option, off by default, makes these actions need the iam:<Action> grant
like every other IAM action: -s3.iam.selfServiceRequiresGrant on weed
server, filer and mini, -iam.selfServiceRequiresGrant on weed s3. With it
set, the grant is authorized against the target user's ARN, the caller's own
when UserName is empty, so a policy can confine it to the caller's own keys
(Resource arn:aws:iam:::user/${aws:username}), the same resource scoping
CreateServiceAccount already uses. The admin and iam:* grants are unchanged;
with the option off nothing changes.
Tests (s3api_embedded_iam_authz_test.go): with the option set, an identity
holding only s3:* and a bare identity are refused all five actions on an
empty, own and other target (fails without the change); a grant on the
caller's own user ARN, literal or ${aws:username}, allows them on itself and
not on another user; the admin and an iam:* grant keep every action; with the
option off self-service is allowed as before.
* iam: wire selfServiceRequiresGrant on the standalone iam server
Every other command that runs the embedded IAM API accepts the flag; the
deprecated weed iam could not enable the enforcement at all.
* iam: wire selfServiceRequiresGrant on weed server and filer -iam
The IamOptions used by 'weed server -iam' and 'weed filer -iam' never
received the flag, so startIamServer dereferenced a nil pointer at
startup. Share the -s3.iam.selfServiceRequiresGrant pointer with them.
---------
Co-authored-by: Javier de la Torre <jatorre@lakehousebox.com>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
* sftp: reject empty passwords in PasswordAuthenticator
FileStore.ValidatePassword already refuses a zero-length stored or
supplied password, but the guard belongs at the authenticator too: an
empty password is never valid, and checking before the store call keeps
any user.Store implementation without its own empty-secret check from
matching on zero-length input.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* sftp: keep the brute-force delay on empty-password failures
The early rejection returned before the shared failure path, so an empty
password failed instantly while other failures slept 100-199 ms. Folding
the length check into the store call keeps the store untouched and every
failed attempt equally slow.
---------
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* s3: keep '\' a literal object-key character when AllowBackslashInKey is set
NormalizeObjectKey rewrites every S3 object key: '\' becomes '/' and
duplicate slashes collapse. On AWS S3 both are ordinary key characters,
so keys that are distinct there alias one object here — writing
probe/back/slash.txt silently overwrites probe/back\slash.txt.
The filer only treats '/' as a path separator, so a verbatim '\' stays
inside one entry name. Gate the backslash rewrite and the matching '\'
handling in IsValidObjectKey behind a process-wide mode the S3 server
sets from the new S3ApiServerOption. Duplicate-slash collapsing stays:
an entry name cannot contain an empty segment.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* command: add -s3.allowBackslashInKey flag for AWS-compatible key handling
Registers the flag on all four S3 entry points (s3, server -s3,
filer -s3, mini) and passes it through S3Options into
S3ApiServerOption. Default off, so existing normalization is
unchanged.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* s3: correct IsValidObjectKey/NormalizeObjectKey docs under allowBackslashInKey
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* s3: normalize DeleteObjects body keys the same as path keys
DeleteObjects keys arrive in the XML body, so they bypassed the
normalization applied to route-var keys. The lock, the authorization
checks, the table-bucket layout check and the filer delete itself all
addressed the raw key while the conditional check resolved the
normalized one, so a key spelled with '\' or '//' could be reported
deleted while the stored entry survived. Echo the client-sent key in
the result while operating on the normalized form.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* s3: report a normalized-empty DeleteObjects key as a delete no-op
A multi-delete key of only slashes normalizes to the empty string. Left
to reach the filer delete, the empty name resolves to the bucket entry
itself, so an unauthorized caller could remove a bucket through the
body-key path. Such a key can never name a stored entry, so like a
missing key the delete is an idempotent no-op reported before any
permission check or filer call.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* s3: drop unused AllowBackslashInObjectKey getter
* s3: answer conditional delete of a normalized-empty key with NoSuchKey
* s3: normalize the probed key in authorizeAnonymousBucketList
The masked-miss probe read the raw mux var, so a spelling like folder//
or folder\ was authorized against a key that can never exist while the
request itself targets the normalized folder/. Probe the normalized key
so the answer matches what the read path authorizes.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* s3: cover anonymous authz of a literal-backslash key under allowBackslashInKey
A public grant on the slash key must not authorize an anonymous read of
the distinct backslash object when the flag keeps '\' literal; grant the
backslash key's own ACL and it reads back. Default mode still folds the
spelling onto the covered slash key.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* s3: publish the key mode only after the S3 server is constructed
A failed NewS3ApiServerWithStore could previously flip the process-wide
key mode out from under an already-running gateway.
* s3: normalize the copy-source key before validation and authorization
pathToBucketObjectAndVersion returned the raw header spelling, so an
equivalent spelling (a//b, a\b) was authorized and compared against the
normalized destination while resolveCopySourceEntry looked up the
normalized key — a deny on the canonical key could be bypassed and
self-copy detection missed. Normalize once at parse so the CopyObject and
UploadPartCopy paths agree.
---------
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* storage: read the super block extra before unmarshaling it
ReadSuperBlock allocated a buffer of ExtraSize bytes but never filled it,
so any volume with a super block extra failed to load with an invalid
wire-format error.
Co-authored-by: chuanying chen <18412672+ccy00808@users.noreply.github.com>
* storage: test super block extra read error paths
---------
Co-authored-by: chuanying chen <18412672+ccy00808@users.noreply.github.com>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
* iceberg: record geometry bounds on the data files compaction writes
Compaction collected column statistics for every primitive column except
geometry: the bound converter has no geometry case, so a compacted file's
manifest entry carried value and null counts for a geometry column but no
lower or upper bound. Readers that skip files by these bounds for a spatial
filter (for example DuckDB for `geom && <box>`) had to open every compacted
file, where the files before compaction could be skipped.
The bound follows the spec's single-value serialization for geometry: the
lower bound is the point (xmin, ymin), the upper bound (xmax, ymax), each an
8-byte little-endian double; z and m are not written. It is the union of the
written file's row-group geospatial statistics, which parquet-go computes
from every non-null value.
A bound that is too small makes a reader skip rows, so any doubt gives none:
a row group with non-null values but no geospatial statistics (parquet-go
omits them when a value does not parse as WKB), a box that is not finite, an
inverted box, or a chunk whose statistics cannot be read. A row group of
nulls only constrains nothing. Geography gets no bound, as iceberg-go writes
none for it (geodesic edges, boxes across the antimeridian).
Tests: points and polygons far apart through a bin-packed compaction (the
bound equals the union of every geometry), a sorted compaction (any bound
covers every geometry), a column of nulls only, an unparseable value in a
second row group (no bound; without the rule the file got the bound of the
first row group only), geography, and the aggregation rules.
* iceberg: do not order compaction bins by a geometry column
When a table declares no usable sort order, compaction orders a bin's files
by the first column every file has bounds for. A v3 table whose first column
is a geometry qualifies as soon as its files carry geometry bounds (DuckDB
writes them, and compacted files now do too). Those bounds decode into geo
literals that compareLiterals cannot compare, so the order sorted nothing,
yet an oversized bin was still split into "ordered" runs by splitOrderedBin.
Geometry and geography columns are now skipped in both the declared sort
order and the inferred one, so such a table is ordered by its next column
with bounds.
Test: TestResolveCompactionOrderSkipsGeometry, a geometry column first and
an id column second; on master the order is the geometry column.
---------
Co-authored-by: Javier de la Torre <jatorre@lakehousebox.com>
* iceberg: authorize a commit before staging its metadata file
Both commit paths (CommitTable and CommitTransaction) write the new
vN.metadata.json into the table's metadata/ directory and only then reach
UpdateTable, whose authorization refuses a caller that may read the table but
not commit to it. The refused commit then deletes the staged file on the way
out, so such a caller could make the server write and delete a file under the
table's prefix on every attempt. The same shape was closed for creates in
#10991.
Both paths now ask Manager.AuthorizeUpdateTable right after loading the table
and answer 403 ForbiddenException before anything is staged. UpdateTable still
decides under the policies in force when the pointer flips, so a policy changed
mid-commit is caught as before.
Test: TestCommitDeniedBeforeStagingMetadata records every file the handlers
create; on master both the single-table and the transaction commit by a
GetTable-only identity write v2.metadata.json before answering 403, with the
change nothing is written. TestCommitAllowedStagesMetadata is the positive
control (the owner, and an identity whose own actions include UpdateTable).
* iceberg: authorize a view commit before staging its metadata
handleUpdateView has the same write-before-authorize gap the table commit
paths had: it stages the new metadata file and only reaches the UpdateView
authorization at the pointer flip, so a caller who may only load the view
could still make the server write and delete files under it. Gate it on
AuthorizeUpdateView, mirroring authorizeCommit.
* iceberg: seed committed view metadata in the denied-commit test
Without stored view metadata the request fails on the missing-metadata
check before it could stage a file, so the zero-write assertion could
not catch a write-before-denial regression. Seed the same metadata the
allowed-path test uses so the denied request would otherwise stage.
---------
Co-authored-by: Javier de la Torre <jatorre@lakehousebox.com>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
* s3: resolve versionId ACL requests to the version ACL actions
A request for an object's ACL carrying versionId acted on the historical version but authorized against the current-object action (s3:GetObjectAcl/s3:PutObjectAcl), because the acl block in resolveFromQueryParameters returned before versionId was considered - the same gap closed for tagging. An identity granted only current-object ACL access - or explicitly denied s3:*ObjectVersionAcl - could still read and modify ACLs on noncurrent versions.
?acl&versionId=<id> on an object now resolves to s3:GetObjectVersionAcl/s3:PutObjectVersionAcl; ?acl alone keeps the current-object actions and bucket ACL is unaffected. s3:PutObjectVersionAcl is mapped onto the WriteAcp identity action so policies can name it.
* s3: resolve versionId attributes requests to the version attributes action
GET /bucket/key?attributes&versionId=<id> fetched the historical version's attributes but authorized against s3:GetObjectAttributes, so a grant limited to the current-object attributes action could reach noncurrent versions. The request now resolves to s3:GetObjectVersionAttributes, and both attributes actions map onto the Read identity action so policies can name them (s3:GetObjectAttributes was previously rejected as an unmapped action).
* s3api: recheck the version action on versioned ACL/attributes requests
The phase-2 policy recheck inside GetObjectAclHandler,
PutObjectAclHandler, and GetObjectAttributesHandler passed the
current-object action even when the request carried versionId and phase 1
had resolved the version action. A Deny scoped to s3:*VersionAcl or
s3:GetObjectVersionAttributes could never fire in the recheck, and an
Allow limited to the version action went unevaluated. Select the action
the same way the resolver does — versioned whenever versionId is present.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* s3api: recheck the version action on versioned tagging requests
The same phase-2 gap exists in the tagging handlers left by the earlier
version-tagging resolution change: versioned GetObjectTagging /
PutObjectTagging / DeleteObjectTagging rechecks evaluated the
current-object tagging action, so a Deny on s3:*VersionTagging could not
fire.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* s3api: resolve HEAD ?acl as the head-object action, not an ACL action
No HEAD acl route exists, so HEAD /bucket/key?acl lands on
HeadObjectHandler — which serves object headers, not the ACL. Resolving
it to an ACL action denied readers who hold only GetObject(Version) and
let ACL grants reach a head-object read. Dropping HEAD from the acl
branch lets the request fall through to GetObject/GetObjectVersion,
matching how the tagging branch already treats HEAD.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* s3api: require the version read grant for versioned attributes
AWS asks for s3:GetObjectVersion alongside s3:GetObjectVersionAttributes
when a GetObjectAttributes request names a versionId; the resolved
version-scoped action on its own left a lone s3:GetObjectVersion Deny
unable to cover the attributes read. Check the version read grant too:
the identity side through VerifyActionPermission, the bucket-policy side
through the same recheck that evaluates the scoped action.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* s3api: check the version read grant on a synthesized versioned GET
The versioned attributes check passed the literal s3:GetObjectVersion
action, but both policy paths re-resolve the action from the request, so
the attributes query parameter mapped it back to
s3:GetObjectVersionAttributes and a lone version-read Deny never fired.
The literal action also reached the legacy CanDo path, where a plain
Read grant no longer matched. Evaluate the grant on a request shaped
like a plain versioned GET instead.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* s3api: reject a named versionId on buckets without versioning
Tagging, ACL, and attributes handlers answered versionId requests on
never-versioned buckets as the current object while evaluating the
version-scoped action, so a grant limited to the version action could
authorize changes to the live entry. Reject the request up front through
the existing checkVersionIdConfigured gate.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* s3api: carry presign params onto synthesized policy requests
objectKeyPolicyRequest rebuilt the query with only versionId and the STS
token, so a presigned request's s3:authType condition value silently
changed from REST-QUERY-STRING to unset during phase-2 rechecks (version
read grant, copy source, delete keys, post-policy writes). A policy
conditioned on s3:authType would then deny a request it already allowed.
Carry the SigV2/SigV4 presign keys through; none of them steer
ResolveS3Action.
---------
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* ec distribution: parse EC and replication configs into a shard plan
NewECConfig/DefaultECConfig and NewReplicationConfig(FromString) turn an
EC ratio and a replica placement into ECConfig/ReplicationConfig values,
and CalculateDistribution derives the target and maximum shard counts per
data center, rack, and node. ECDistribution.Summary and
FaultToleranceAnalysis describe, per topology level, whether losing one
failure domain leaves enough shards to reconstruct the volume, so the EC
durability guarantees of a given setup can be answered directly instead of
only documented informally.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* ec distribution: analyze topology and plan shard moves
TopologyAnalysis records nodes and shard locations per DC/rack/node, and
Rebalancer.PlanRebalance emits the moves that level the layout: data
centers first, then racks within each DC, then nodes within each rack,
moving parity shards before data shards and honoring per-node free slots.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* ec distribution: cover the plan math and rebalancing with tests
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* ec distribution: keep AddShardLocation and node ShardIDs in sync
AddShardLocation updated the count maps but never the node's ShardIDs —
yet every rebalancer count reads len(node.ShardIDs), so the documented
AddNode-then-AddShardLocation flow produced an empty rebalance plan. It
now appends to the node's shard list and skips shards the node already
has, making the two registration styles equivalent and idempotent.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* ec distribution: key rack-scoped maps by dc/rack
Rack names are only unique within a data center — two DCs can both hold a
"rack1" — so ShardsByRack, RackToShards, and RackToNodes keyed by the bare
rack name merged distinct domains and corrupted both the DC-level shard
counts and rack-level balancing. Rack lookups now use the dc/rack pair;
DCToRacks still lists bare rack names.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* ec distribution: never move a shard onto a node already holding it
The move picker selected the destination node before the shard, so a node
already holding the chosen shard could be targeted. Pick the shard first
(parity-first) and choose among destination nodes that lack it.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* ec distribution: tolerate zero-value replication configs
A ReplicationConfig{} with zeroed minimums made maxShare divide by zero;
clamp each level's unit count to at least one. Covers the new behaviors
with tests.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* ec distribution: keep rack keys distinct when names contain "/"
rackKey joined data center and rack with "/", so dc "a"/rack "b/c" and
dc "a/b"/rack "c" shared one key and were counted as the same failure
domain. Separate the parts with a NUL byte, which cannot appear in a
name.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* ec distribution: plan rebalances on a copy of the topology
recordMove mutated the caller's TopologyAnalysis while planning, so an
interrupted or partial execution left the input describing moves that
never happened, and replanning on it returned nothing. PlanRebalance
now simulates on a cloned analysis; plan.Analysis reports the layout
the moves would produce and the caller's input is left untouched.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* ec distribution: balance nodes against the rack's current shard count
The node-level target still read the original analysis, so a rack that
had just sent or received shards during DC/rack balancing aimed at a
stale total and left its nodes unbalanced.
* ec distribution: register each node id once
Adding the same node twice appended it to RackToNodes twice, so every
count that walks rack nodes double-counted its shards.
* ec distribution: normalize replication counts in the stored plan
A zero-value ReplicationConfig computed targets against one domain per
level while Summary still reported zero DCs, racks, and nodes.
* ec distribution: cap a survivable level's max at the loss budget
* ec distribution: skip shards already inside the destination domain
* ec distribution: register nodes implied by shard locations
* ec distribution: merge repeat node registrations
A shard location now auto-registers an unknown node with no free slots;
a later AddNode carrying the real capacity was then dropped entirely.
Repeat registration now updates the node's capacity and adopts any new
shards, while the first registration keeps the topology placement.
* ec distribution: try other overloaded sources before stopping
When the most overloaded domain could not shed any shard, balance
returned early and skipped every other source that still had movable
shards. A source that cannot move now gets set aside for the rest of
the pass.
* ec distribution: collapse chained shard moves to the final destination
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* ec distribution: schedule collapsed moves behind the departures that free slots
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
The max volume count subtracted each volume's remaining room from the
free disk space, except when the master had preallocation turned on,
in which case it subtracted nothing. That is wrong for a disk that
mixes preallocated and plain volumes, and on platforms where
preallocation is not supported.
Use the larger of the .dat file's size and its allocated blocks, in the
Go and the Rust volume server, so a preallocated volume counts as full
and a plain one keeps its remaining room reserved.
Fixes#5778
Co-authored-by: Dan Foster <4570306+danfoster@users.noreply.github.com>
* iceberg: keep a default-CRS geometry column without a CRS after compaction
Writers such as DuckDB write a GEOMETRY column whose Iceberg type has the
default CRS (OGC:CRS84) with no CRS in the Parquet logical type. parquet-go
spells the default out when it writes such a column (geometryType's
LogicalType), even when the column it read declared none, so every
compacted file declared GEOMETRY(crs=OGC:CRS84) where its inputs declared
GEOMETRY. Both mean the same to the Parquet spec, but a reader may treat the
file column as another type than the table column: DuckDB casts it to the
table's GEOMETRY, and a cast column is no longer pruned by its row-group
statistics, so a spatial filter read every row group of a compacted file.
After a bin is merged, the footer's schema elements of every GEOMETRY
column whose Iceberg type has the default CRS (top-level or nested) are
rewritten to declare no CRS. The data, the page indexes and every offset
stay byte for byte; only the footer is re-encoded. An explicit CRS is left
as it is. Retyping the writer's column instead does not work: parquet-go
refuses to convert between two GEOMETRY types that both mean the default,
and its sorting writer converts.
Tests: geometry_crs_test.go, a bin-packed and a sorted compaction of
default-CRS inputs (one of them spelling the default out) write no CRS,
with a control showing parquet-go spells it out on its own; an explicit CRS
is kept; a nested geometry. On master the binpack and sort cases fail with
CRS "OGC:CRS84".
* iceberg: compress the row lineage columns of compacted v3 files
A v3 compaction appends _row_id and _last_updated_sequence_number to the
merged file. The two leaves were built without a codec, so they were
written uncompressed, 16 bytes a row, whatever codec the data columns use;
on a table of points the compacted files grew by about a third.
The lineage leaves now take the codec of the reference file's first leaf
that declares one, ZSTD when none does.
Test: TestCompactCompressesLineageColumns; on master both columns are
written UNCOMPRESSED next to ZSTD data columns.
* iceberg: match list and map geometry by Iceberg name in files without field ids
A Parquet footer path runs through the repeated group of a LIST or MAP
(shapes.list.element, attrs.key_value.value), which Iceberg leaves out of
its field names (shapes.element, attrs.value). The name lookup for a file
without field ids therefore missed every geometry inside a list or a map
and left the default CRS spelled out. Name the paths as Iceberg does,
including the legacy two-level list forms.
* iceberg: rewrite the geometry CRS footer inside the merged file's buffer
Removing a CRS shortens the footer, so the new one fits where the old one
was. Copying the whole merged file to replace a few hundred bytes of
footer doubled the memory of a compaction near the target file size.
* iceberg: take the lineage codec from the data columns, not an input's _row_id
An input that already carries _row_id may list it before its data
columns, and write it uncompressed; the new lineage columns then copied
that codec instead of the data columns'. Choose from the columns the
layout keeps.
---------
Co-authored-by: Javier de la Torre <jatorre@lakehousebox.com>
* s3tables: accept the external-volume verification object at a table bucket's root
Snowflake verifies an S3-compatible external volume before it creates or
writes Iceberg tables in it (SYSTEM$VERIFY_EXTERNAL_VOLUME, also run
implicitly): it writes, reads, lists and deletes one small object,
<base>/verify_<epoch milliseconds>_<number>, a 21-byte body. A table bucket
refuses every key outside the Iceberg layout, so the check failed and
Snowflake could not use a table bucket as an external volume.
PUT, GET, HEAD, DELETE and DeleteObjects now accept exactly that key in a
table bucket, narrowly enough that it cannot be used to store data:
- at the bucket root only, so nothing is encoded in a prefix, written into
a hidden tree or placed on a namespace or table name;
- one shape: verify_<13 digits of epoch ms within one hour of the server's
clock>_<1-10 digits>;
- a PUT must declare a Content-Length of at most 1 KiB (no chunked or
undeclared body) and carry no user metadata, tagging, Content-Disposition,
Cache-Control, Expires or website redirect.
Multipart, copy, rename, ACL, tagging and POST policy writes keep the plain
layout rule. Buckets that are not table buckets are unaffected.
Tests: table_bucket_verify_object_test.go, the key accepted (with the
Iceberg layout as the positive control and the plain rule still refusing
it), 15 look-alikes refused (prefixed, hidden tree, wrong shape, outside the
window, ordinary keys), and seven PUT refusals through PutObjectHandler.
* s3tables: keep the verification key off namespaces and narrow its PUT
A namespace may be named like a verification key (verify_<ms>_<n>), and
the key then names the namespace's directory. A PUT replaced that
directory with a prefix object, dropping the namespace's catalog
metadata; a DELETE removed an empty namespace. The key now gets the plain
layout rule when it names a directory or its entry cannot be read, and a
PUT of it never falls back to a prefix object, so a namespace created
after the check is not replaced either.
The time window applies to PUT only: GET, HEAD and DELETE take a key of
any date, so an object a check left behind for more than an hour can
still be read and removed. A verification PUT also refuses an extraction
request and the stored Content-Language, Response-Content-Disposition and
Content-Encoding headers (aws-chunked framing alone is still accepted).
The integration test runs the check's PUT, HEAD, GET, list and DELETE
against a table bucket, and the PUT and DELETE over a namespace of the
key's shape.
---------
Co-authored-by: Javier de la Torre <jatorre@lakehousebox.com>
* s3 backend: optional HTTP client settings for the cloud tier
The S3 tier backend builds its client from a bare aws.Config, so it always
uses Go's default transport: HTTP/2 wherever the endpoint offers it over TLS,
no cap on connections per host, and a 90s idle timeout. Against some
S3-compatible services that is a poor fit. Over HTTP/2 every upload and
range read of a tiered volume shares one connection and its flow-control
window; with a small window, tier uploads and concurrent reads of tiered
volumes become throughput-bound on that one connection, while HTTP/1.1 opens
a connection per concurrent request. Some services also limit parallel
connections per client address, or drop idle connections before 90s.
Add three optional settings to [storage.backend.s3.<id>], all off by
default so existing deployments keep the SDK's default client:
- disable_http2: keep the client on HTTP/1.1 (ForceAttemptHTTP2=false and an
empty TLSNextProto, the documented opt-out);
- max_connections: MaxConnsPerHost, 0 = no cap;
- idle_connection_timeout_seconds: IdleConnTimeout, 0 = Go's default.
The session cache key includes the options when any is set, so two backends
on the same endpoint with different settings do not share a client.
ToProperties reports the settings; the master.toml scaffold documents them.
Tests: TestS3BackendHTTPOptionsReachTheClient fails without the change (the
settings never reach the client); TestS3BackendDefaultHTTPClientUnchanged,
TestS3BackendSessionsDifferByHTTPOptions, TestParseNonNegative.
* s3 backend: reject an idle timeout too large for time.Duration
An idle_connection_timeout_seconds above math.MaxInt64/1e9 overflowed when
multiplied by time.Second. The negative result made the transport keep its
default timeout and ToProperties export a negative value. Warn and use the
default instead.
* s3 backend: parse the idle timeout as 64-bit seconds
parseIdleConnTimeout went through strconv.Atoi, so on 32-bit platforms
any value above 2147483647 seconds was rejected while the same value was
accepted on 64-bit, and the overflow test failed under GOARCH=386. Parse
with ParseInt(…, 64) and export the effective value with FormatInt so the
accepted range is the same on every platform.
---------
Co-authored-by: Javier de la Torre <jatorre@lakehousebox.com>
s3api: exercise the inline-content route through CopyObjectPartHandler
Generated with [Devin](https://devin.ai)
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* s3: answer InvalidBucketName to a denied request on a malformed bucket name
The store authorizes a request before it looks at the bucket, so a caller
denied on a bucket whose name breaks the S3 naming rules (upper case, an
underscore, too short, two adjacent periods, a non-ASCII letter) gets 403
AccessDenied, where AWS S3 answers 400 InvalidBucketName. Clients and
compatibility suites that check the error code read the 403 as a permission
problem.
Auth and AuthPostPolicy now turn an AccessDenied for a resolved identity into
InvalidBucketName when the bucket name fails VerifyS3BucketName or uses a
character outside a-z, 0-9, '.' and '-' (VerifyS3BucketName checks with
unicode.IsLower/IsNumber, which accept letters such as 'ü'). The answer depends
on the name alone, so it reveals nothing about which buckets exist. An
authentication failure, an allowed request and a denial on a well-formed name
answer as before.
Test: TestAuthDeniedInvalidBucketName drives iam.Auth with signed and anonymous
requests; on master every malformed name answers 403, with the change 400,
while the identity's own bucket stays allowed and a well-formed foreign name
stays 403. TestRefineDenialForBucketName covers the rule table.
* s3: refine POST-policy and deferred anonymous-read denials on a malformed bucket name
Two denial paths bypass the refinement in Auth and AuthPostPolicy:
- A browser POST upload carries its credentials in the form, so
AuthPostPolicy delegates it to PostPolicyBucketHandler, which verifies the
form and writes AccessDenied itself when the signer may not write the bucket
or the per-key authorization denies the object. Both denials now go through
refineDenialForBucketName with the verified identity.
The coarse Write check used to run inside doesPolicySignatureV2Match and
doesPolicySignatureV4Match before the signature was compared, so a form with
a wrong signature was told whether the key may write the bucket. It now runs
in doesPolicySignatureMatch after the signature verifies, as on the
header-signed path; a form that fails verification answers
SignatureDoesNotMatch (or MissingFields) and is never refined.
- Unsigned GET/HEAD object reads are deferred by AuthWithPublicRead to the
object handlers, where writeAnonymousAwareNotFound masks a miss as
AccessDenied. That denial is refined with the anonymous identity, so a
missing malformed bucket answers InvalidBucketName when an anonymous
identity is configured, as Auth does. A denial on an entry that was found
is left alone: the bucket exists.
Test: TestPostPolicyDeniedInvalidBucketName drives AuthPostPolicy with signed
forms (Write denied, per-key policy deny, bad signature, well-formed name);
TestAnonymousReadDeniedInvalidBucketName drives AuthWithPublicRead with GET and
HEAD, with and without an anonymous identity. Both fail before this change.
* s3: refine a denied anonymous read of an existing key on a malformed bucket name
A deferred anonymous GET/HEAD refined only its masked miss to
InvalidBucketName, so in a bucket whose name breaks the S3 rules (one
VerifyS3BucketName still accepts, such as a non-ASCII name) a missing key
answered 400 and an existing private key 403, telling the caller which
keys exist. The entry check now refines its denial the same way, so both
answer alike. Without an anonymous identity no identity resolves and both
stay 403, as in Auth.
The deny-precedence case in TestAnonymousObjectACLConfiguredIdentity runs
on the fixture bucket "b", shorter than S3's three-character minimum, so
its denial now answers InvalidBucketName.
---------
Co-authored-by: Javier de la Torre <jatorre@lakehousebox.com>
An out-of-band empty directory listed under its own trailing-slash
prefix answered the probe with a fake entry of itself: a Contents key on
4.48 and, since recently, a CommonPrefix. That listing enumerates
children, where S3 returns nothing for an empty prefix, and clients that
read every returned key (Pulumi lock listing) choked on the marker.
The prefix object and directory key object paths are unchanged: a real
"dir/" object still lists as its own key.
Generated with [Devin](https://devin.ai)
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* volume tier: accept deletes on a volume as soon as it is tiered
volume.tier.upload marks the volume read-only without deletes before the
upload (markVolumeReplicasWritable(..., false, false), i.e. noWriteOrDelete),
and nothing clears that until the volume server restarts, when load() gives
every tiered volume noWriteCanDelete. Every delete in between is refused with
"volume N is read only": the needle stays live in the index, and its bytes in
the remote object, until someone notices.
After the remote file is loaded, the upload handler now gives the volume the
flags a restart would (MarkTieredDeletable: writes refused, deletes accepted),
with two exceptions that keep the current behaviour:
- a replicated volume: the shell copies the uploaded replica's .idx and .vif
to the other replicas after the upload, and a tombstone written before that
copy would reach one replica only;
- a volume in I/O quarantine: that flag is not the upload's to lift.
Tests: TestTierUploadAcceptsDeletesOnASingleCopyVolume fails without the
change ("volume 1 is read only") and checks writes stay refused;
TestTierUploadKeepsRefusingDeletesOnAReplicatedVolume; TestMarkTieredDeletable
(quarantine kept).
* volume tier: check the I/O quarantine under noWriteLock
MarkTieredDeletable read the quarantine state before taking noWriteLock.
CollectHeartbeat could quarantine the volume and set noWriteOrDelete in
between, and the upload then cleared that refusal, so a quarantined volume
accepted deletes until the next heartbeat. Take noWriteLock first: the
heartbeat marks the quarantine before it takes noWriteLock, so either the
check sees it or the heartbeat's refusal lands afterwards and stays.
Also leave alone a volume past IoErrorTolerance that the next heartbeat
will quarantine, as the heartbeat itself does.
* volume: read noWriteOrDelete under noWriteLock in DeleteVolumeNeedle
DeleteVolumeNeedle checked v.noWriteOrDelete without the lock that
MarkVolumeReadonly, CollectHeartbeat and now MarkTieredDeletable hold when
they write it, so a delete racing a tier upload was a data race. Copy the
flag under the read lock; the delete path is otherwise unchanged.
* volume tier: mirror the delete acceptance rules in the Rust volume server
The Rust tier upload already promoted a freshly tiered volume to
no-write-can-delete via refresh_remote_write_mode, and did so for every
volume -- including the replicated and I/O-quarantined ones the Go
MarkTieredDeletable deliberately refuses. A live tier-up now goes
through mark_tiered_deletable (replicated volumes and quarantines keep
their upload-time mark until the next load), while load() keeps
promoting every tiered volume unconditionally, as Go does.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* volume: keep a tiered volume delete-only when marked writable
MarkVolumeWritable preserved noWriteCanDelete on a remote-tiered volume
only when it was already set. A tiered volume still holding the
upload-time mark (a replicated volume, or one whose tier upload ran
before MarkTieredDeletable existed) had it clear, so marking writable
published the volume as fully writable while its .dat lives remotely --
writes then failed against a backend that cannot append, after the
master was told the volume is writable again.
load() gives every remote-tiered volume noWriteCanDelete; deriving the
same flag here keeps mark-writable consistent with a restart: "writable"
on a remote volume resolves to can-delete.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* volume: keep a tiered volume delete-only when marked writable (rust)
set_writable preserved the remote delete-only mode only when the flag was
already set, so a tiered volume holding the upload-time read-only mark
(replicated, or quarantined under mark_tiered_deletable) was published
fully writable with its .dat remote. Derive no_write_can_delete from
has_remote_file() instead, matching load() and the Go MarkVolumeWritable.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* volume tier: serialize the quarantine check with the flag swap (rust)
A heartbeat refusal landing between should_quarantine() and the store
could be clobbered until the next heartbeat re-asserted it — the same
race the Go side closed by checking under noWriteLock. A
compare-exchange gives the same ordering: the swap fails, and the retry
sees the quarantine mark the heartbeat sets first.
* volume tier: MarkTieredDeletable refuses a non-tiered volume
A stray call on a local single-copy volume would pin it
read-only-can-delete. The only caller promotes right after
LoadRemoteFile, but the guard makes the method safe to reuse; the Rust
mark_tiered_deletable gets the same check, though its caller already
only reaches it under has_remote.
---------
Co-authored-by: Javier de la Torre <jatorre@lakehousebox.com>
Co-authored-by: chrislusf <chris.lu@gmail.com>
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* s3tables, iceberg: log client refusals at V(1), not ERROR
Requests the S3 Tables and Iceberg REST APIs correctly refuse with a 4xx
(access denied, not authorized, no such namespace, already exists, not
empty, a commit conflict) were logged with glog.Errorf, so a client probing
for what it may do, or retrying an optimistic commit, filled the ERROR log
and hid real faults from anyone alerting on it.
- s3tables: HandleRequest wraps the ResponseWriter in a statusRecorder (with
Unwrap for http.ResponseController) and logs a handler error by the status
it answered: 4xx at V(1) as "<op> refused (<status>)", 5xx or no status at
ERROR as before.
- s3tables: the "handleCreateNamespace called" trace moves from ERROR to V(3).
- iceberg: writeManagerError's mapping is split into classifyManagerError,
shared with a new logManagerError that logs 4xx-mapped errors at V(1) and
everything else at ERROR; used by CreateNamespace, CommitTable and
CommitTransaction.
- storage: loading a remote-tier volume logged
"readSuperBlock remote volume N: <nil>" at ERROR even when the read
succeeded; it is now logged only when there is an error.
Tests: TestRequestRefusedWith4xxIsNotLoggedAsError, TestLogRequestErrorByStatus,
TestStatusRecorderKeepsTheFirstStatus, TestStatusRecorderUnwrapsForResponseController,
TestNoTraceLoggedAtErrorLevel (fails on master on the CreateNamespace trace),
TestLogManagerErrorRefusalsAreNotErrors, TestRemoteSuperBlockLogOnlyOnError.
* s3tables: do not log a malformed CreateNamespace body at ERROR
handleCreateNamespace logged a body it could not decode at ERROR before
answering 400; the dispatcher already logs that refusal once at V(1).
Also drop statusRecorder.Flush: it made the recorder an http.Flusher even
over a writer that cannot flush, so http.NewResponseController reported
success instead of ErrNotSupported. Unwrap already lets the controller
reach the wrapped writer.
* iceberg: require exactly one log line per classified manager error
The test accepted a refusal that produced no log, or both logs. Require
one V(1) refusal and no ERROR for a 4xx, and the reverse for a fault.
---------
Co-authored-by: Javier de la Torre <jatorre@lakehousebox.com>
* s3api: regression test for UploadPartCopy ETag on ranged copies
Generated with [Devin](https://devin.ai)\n\nCo-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* s3api: hash the copied bytes for a partial-chunk UploadPartCopy ETag
A ranged UploadPartCopy that lands inside a source chunk stored the source chunk's ETag on the destination chunk, so the part reported the MD5 of the wrong bytes. streamCopyChunkRange now tees the forwarded plaintext into an optional digest writer; copySingleChunkForRange feeds it an md5 only when the range is not the whole source chunk, since a whole-chunk copy's stored bytes (and ETag) are unchanged.\n\nGenerated with [Devin](https://devin.ai)\n\nCo-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* s3api: stamp the copied range's MD5 on UploadPartCopy part entries
The part entry got no Attributes.Md5, so copyEntryETag fell back to filer.ETagChunks: a single-chunk part inherited the source chunk's MD5 and a multi-chunk part reported a <md5>-N composite of source chunks. copyChunksForRange now returns the MD5 of the copied bytes — a single chunk's own digest, or for multi-chunk parts a second ordered stream of the source range hashed beside the concurrent chunk copies (md5 cannot combine per-chunk digests). The handler stores it on Attributes.Md5 and mirrors putToFiler's ExtETagKey, so CopyPartResult, ListParts and the completed object's composite ETag use each part's own MD5 (#11855).\n\nGenerated with [Devin](https://devin.ai)\n\nCo-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* s3api: drop the source chunk's gzip flag on partial-range copies
A ranged UploadPartCopy fetches raw (decompressed) bytes from the source volume and stores them uncompressed, but the destination chunk kept the source's IsCompressed flag, so reads of a partial copy of a gzip-stored chunk would try to decompress raw bytes.\n\nGenerated with [Devin](https://devin.ai)\n\nCo-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* s3api: hash the chunk range, not the entry, for the part MD5
getEncryptedStreamFromVolumes prefers inline Content over chunks, but copyChunksForRange copies the chunks — on an entry carrying both, the hash would describe different bytes than the copy.\n\nGenerated with [Devin](https://devin.ai)\n\nCo-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* s3api: cover sparse ranges in the part MD5 and fail on hash errors
A single chunk's digest is the part MD5 only when it supplies the whole range; a sparse source zero-fills the rest, so gaps are hashed via the ordered range stream, and a fully chunkless range hashes zeros locally. A failed hash pass now fails the part copy instead of storing a chunk-composite ETag that describes the wrong bytes.\n\nGenerated with [Devin](https://devin.ai)\n\nCo-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* s3api: rank copied-range destination chunks by source timestamps
The concurrent chunk copies stamped each destination chunk with an arbitrary finish time, but reads resolve overlapping chunks by ModifiedTsNs — a copy of overlapping source chunks could expose different bytes than the source (and than the part MD5 just hashed). Destination chunks now get timestamps ranked by the source's, preserving its overlap precedence.\n\nGenerated with [Devin](https://devin.ai)\n\nCo-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* s3api: cancel the part-MD5 stream when a chunk copy fails
The hashing goroutine ran on context.Background(), so an early chunk-copy error return left it streaming and hashing the whole source range for a discarded result. A cancellable context now aborts it on every exit path.\n\nGenerated with [Devin](https://devin.ai)\n\nCo-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* s3api: hash the copied range when the covering chunk has no usable ETag
TUS and mount writes leave chunks without an ETag, so a whole-range
UploadPartCopy would otherwise save the part with an empty Md5 and no
ExtETagKey, and multipart completion rejects the part.
* s3api: build the md5 source entry without copying the proto message
* s3api: drop the issue reference from the part-copy etag test
* s3api: route inline-content sources through the plaintext part copy
The raw-byte path walks entry.GetChunks() only, so a ranged
UploadPartCopy of an inline-Content source (filer HTTP or mount writes)
produced a zero-filled part. Send those through
copyObjectPartViaReencryption, which streams the real bytes and stamps
their MD5 on the part.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* s3api: store the empty-part MD5 on writeEmptyCopyPart entries
The slow path's empty part returned the right CopyPartResult ETag but
saved no Attributes.Md5 or ExtETagKey, so ListParts and the completed
object's composite ETag still fell back to a chunk-derived value.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* s3: apply Range to HeadObject
* s3: apply Range to a HEAD of a stored directory key
Directory-key entries answer through serveDirectoryContent before
HeadObjectHandler's range branch, so a ranged HEAD returned 200 and the
full length.
* s3: keep the missing-directory check for plain directories only
A stored directory-key marker is an object, so a range on its empty body
answers 416 like on any other zero-byte object.
* s3: let partNumber keep its priority over Range on a directory key HEAD
* s3: serve ranged GETs of a stored directory key
The same branch now applies the validated range to the inline content on
GET, so a directory-key object answers 206 with the selected bytes like
any other object.
---------
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
* volume: route lookups to the leader master only (#11887)
After a leader change, the volume server kept sending volume and EC
shard lookups to masters that were not the leader:
* currentMaster was set as soon as a heartbeat stream opened, before
the peer said whether it was leader, and kept a follower through the
redirect and its 3s sleep. A follower answers LookupVolume from its
own vid cache, which right after an election can be empty, so reads
went 404 rather than failing over.
* EC lookups used Store.MasterAddress, a second, unlocked copy that the
heartbeat loop set before each dial and cleared on errors, so EC and
plain reads could disagree on the master or have none.
* the receive goroutine called stream.Send itself when the volume
options changed, racing the send loop, and exited silently on
failure, after which leader redirects were no longer read.
Publish currentMaster only once a master is known to be leader: the
peer answered a new stream without naming a leader, which the leader
always does, or a follower named the leader. Compare redirects against
the connected peer. Replace Store.MasterAddress with MasterFn, backed
by getCurrentMaster. Have the receiver ask the send loop for the extra
heartbeat instead of sending it.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* volume: ask the masters for the leader after losing the heartbeat
When the heartbeat stream to the leader failed, currentMaster kept the
lost master until a new stream answered, so lookups on a cache miss
kept dialing it through the retry delay even with a new leader already
elected.
On a regular heartbeat error, ask the configured masters for the raft
leader and publish it, within the retry delay rather than on top of it.
Without an answer the last known leader stays: clearing it would only
trade a dead address for an empty one.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* test: stop the heartbeat on a failed assertion
The leader test closed stopChan only once every assertion had passed,
so a failure left the stream open and the fake master's GracefulStop
waited for the whole test run's timeout.
Start the heartbeat through a helper whose cleanup stops it, safe to
call twice, and waits for it to exit. It runs before the cleanups
registered earlier, so the stream is closed before the store and the
fake master go away. Both heartbeat tests use it.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* volume: clear the leader when no master names one
When the heartbeat lost its master and no configured master named a
leader, refreshLeaderFromPeers kept the lost one. That master is dead
or no longer leader, and a follower answers LookupVolume from its own
cache instead of refusing, so lookups could return locations no leader
vouched for.
Clear currentMaster instead. Lookups then fail fast ("missing address")
until a leader is known again, rather than reading a follower's cache
or waiting on a dial to a dead master.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* volume: keep volumeSizeLimit 64-bit aligned on 32-bit builds
MasterFn (a func, 4 bytes on 386) replaced MasterAddress (a string, 8
bytes) ahead of volumeSizeLimit and moved it to a 4-byte-aligned
offset, where atomic.LoadUint64 panics with an unaligned 64-bit
operation. The atomic.Uint64 type carries the alignment itself.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* volume: publish the answering leader before a duplicate-uuid retry
Read the leader field first on each heartbeat reply. A follower reply
carries nothing else, so it no longer runs through the option updates;
and a leader that rejects the registration over duplicate uuids still
answers lookups, so it stays the lookup target through the retry wait
instead of leaving the previous master in place.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* volume: publish the master only once it is known leader (rust)
* volume: ask the masters for the leader after losing the heartbeat (rust)
* volume: read the leader field first on each heartbeat reply (rust)
* volume: drop the seed fallback from ec shard lookups (rust)
cached_lookup_ec_shard_locations fell back to the static first seed when
no leader was published, so a mid-election EC lookup could land on a
follower and read its stale shard map. Error out like Go's
masterAddress() returning "" instead, via a current_master() getter
matching getCurrentMaster.
* volume: send volume lookups to the confirmed leader, not the seed (rust)
Every HTTP lookup path — proxy/redirect targets, replicated writes and
deletes, batch chunk deletes, remote chunk reads — dialed the static
first seed, so after a failover they kept asking a follower whose vid
map can be stale. Go routes all of these through GetMaster; switch the
lookup calls and their leader-known gates to current_master().
* volume: mark volumes readonly on the leader, retrying the named one (rust)
notify_master_volume_readonly fell back to the seed when no leader was
published, which the peer answers with "Not current leader". It now
uses only the confirmed leader, skips the call in masterless mode, and
on NotLeader asks the seed masters who leads and retries there — the
same path Go's notifyMasterVolumeReadonly takes. The peer leader query
is shared with refresh_leader_from_peers as lookup_leader_from_peers.
* volume: ask the confirmed leader for copy prealloc options (rust)
VolumeCopy's preallocate query went to the static seed; Go issues it
through GetMaster. Route it through current_master() too — when no
leader is known the lookup fails fast and the copy proceeds without
preallocating, as before.
* volume: keep the announced master port forms, not just http (rust)
publish_current_master stripped to http form, so a leader configured
with an explicit gRPC port (host:9333.19999) was redialed on the
default offset. Store the announced host:port[.grpcPort] form like Go's
currentMaster; callers pick to_http_address / to_grpc_address /
parse_grpc_address per use, and Ping admission compares on the http
form.
* volume: fail replicated writes and deletes when no leader is known (rust)
Gating replication on a known leader meant a write could be
acknowledged on fewer copies than required, and a delete could leave
the data readable on the other replicas. Go fails the request when the
replica-location lookup cannot run; drop the gate so the empty-leader
lookup errors the same way.
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
* s3tables: judge an absent tag target by its canonical ARN
authorizeAbsentTarget matched the bucket policy's Deny statements
against the request's raw resource ARN. The ARN parsers accept any
account and region but resolve the target by name, so a caller with an
identity grant could reformat the ARN, miss the Deny match, and learn
whether a denied name exists — the oracle the absent-target check was
meant to close. Rebuild the ARN from the resolved names and the real
bucket owner and evaluate the Deny against that, matching what the
existing-target path checks.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* s3tables: cover a foreign-account ARN on an absent denied table
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* s3tables: keep the absent-target deny check behind a supplied ARN
Callers that pass no resource ARN (ListTables, ListViews,
CreateNamespace) must reach CheckTableOrBucketPermission's admin
shortcut unchanged; running the bucket-policy Deny match first would
deny the admin lists those callers take.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(s3): set Access-Control-Allow-Credentials on CORS responses (fixes#11885)
* fix(s3): expand wildcard exposed headers for credentialed CORS responses
* fix(s3): keep wildcard in exposed headers for non-credentialed CORS requests
* fix(s3): expose x-amz-tagging-count on credentialed CORS responses
---------
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
* s3tables: authorize before revealing a list or tag target exists
ListTables, ListViews and the tag operations read the target's catalog
metadata before any permission check, so a denied caller could tell an
existing name (403 or the resource's own error) from an absent one (an
empty list, or 404). Evaluate the operation's permission with whatever
bucket context could be read and answer a denied caller the same way for
absent, damaged and denied targets.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* s3tables: answer bucket reads uniformly when the entry is not a table bucket
A plain S3 bucket directory answers GetTableBucket and ListNamespaces with
500 attribute-not-found while a missing name gets 404, and a denied
CreateNamespace is 403 while a missing bucket is 404. Run the permission
check before the absence answer and read not-a-table-bucket as
NoSuchTableBucket.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* s3tables: cover absent-target answers matching denied ones
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* iam: take the implicit user name from the verified identity
For self-service actions sent without UserName, the implicit user was
resolved by re-parsing the raw Authorization header for a SigV4
credential and looking that access key up. The header is not what
authentication verified: a request can be authenticated by one
mechanism while carrying a header that names another user's access
key, and the action then ran against that other user.
The auth middleware already records the verified identity in the
request context, so use it, on both the standalone iam server and the
embedded IAM surface. When no verified identity is present the name is
left unset rather than guessed.
* iam: test implicit user name resolution
Cover a request authenticated by presigned credentials that also
carries a junk SigV4 header naming another user: the self-service
actions must run as the verified signer. Update the unit tests for the
context-based lookup.
* iam: resolve the implicit user for standalone GetUser too
The embedded surface fills UserName for GetUser; the standalone server
skipped it, so an implicit GetUser on `weed iam` answered NoSuchEntity.
Same verified-identity source as the other self-service actions.
* iam: cover all implicit-username actions in the regression test
Table and partition statistics are Puffin files that the table metadata lists
in its statistics entries; orphan cleanup keeps them (#11651), and its test
places one at metadata/stats-1.puffin. The table bucket layout rule accepted
only *.stats among them, so an engine that names a statistics file *.puffin
had the upload refused with 403 and could not write statistics at all.
The metadata patterns now admit a *.puffin file directly under metadata/,
with the same safe-character rule as the other catch-all patterns. Nothing
else widens: a Puffin file in a subdirectory of metadata/, with a trailing
extension, or outside <namespace>/<table>/ is still refused.
Test: the new metadata/*.puffin cases in
TestIcebergLayoutValidator_AcceptsRealWorldManifestNames and
TestTableBucketFileValidator_PuffinStatisticsOnlyInTableMetadata fail on
master ("invalid metadata file format") and pass with the change; the
rejected cases pass on both.
Co-authored-by: Javier de la Torre <jatorre@lakehousebox.com>
* filer.remote.sync: do not upload an entry whose local copy was dropped
remote.uncache and the filer's eviction under storage pressure drop a
synced entry's chunks and clear LastLocalSyncTsNs; remote.meta.sync does
the same when it pulls a newer remote version. When the remote mtime
falls in the entry's mtime second, which is the common case for a small
file synced promptly, shouldSendToRemote falls back to the cleared stamp
and reads the entry as dirty. The dropped chunks make the update a
content change, so the chunkless entry was uploaded and a zero-byte
object replaced the remote one, the entry's only content.
Skip an update that leaves the entry at the same key with no local data
and a sized RemoteEntry: the object at that key already is the content.
The stamp stays cleared, so remote.cache and eviction still see the
entry as uncached. Renames keep going through completeRemoteOnlyRename,
metadata-only edits of a remote-only entry still reach the remote, and a
rewrite in the remote mtime's second holds local data and is uploaded.
* filer.remote.sync: upload a truncate that leaves the entry remote-only
A FUSE truncate of a cached, remote-backed file to zero (truncateEntry,
or SetAttr with size 0) drops the chunks and keeps the sized
RemoteEntry, so the entry reads as remote-only and the dropped-copy skip
left the remote object with its old content.
Skip only while the entry's size is still the remote object's. Uncache
and eviction keep the size and remote.meta.sync sets it from the
object, so those stay skipped; a truncate changes the size and is
uploaded, in the remote mtime's second too.
- Go/Rust: run the unindexed .dat tail repair on noWriteCanDelete volumes
too — deletes still append tombstone records, so the same crash window
applies and they previously degraded to fully read-only.
- Go readTailNeedle: reject a positive size of 1-3 for v2+ as corrupted
before ReadBytes slices the body's 4-byte data-size field and panics;
a torn last record is now cut instead of crashing the load.
- Tests: TestVolumeLoadRepairsTailOnCanDeleteVolume and
TestVolumeLoadCutsDamagedSmallSizeTailOnCanDeleteVolume (Go), and the
Rust mirrors volume_load_repairs_tail_on_can_delete_volume /
volume_load_cuts_damaged_small_size_tail.
Generated with [Devin](https://devin.ai)
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* rust volume: repair an unindexed .dat tail at load
Mirror Go's repairUnindexedDatTail (#11870): a crash between the .dat\nappend and the .idx append leaves a .dat tail no index entry covers,\nwhich sent the volume read-only on the next load and left acknowledged\nneedles unreadable. Before the index loads, re-index the complete\nneedles in the tail (a size 0 record as the tombstone or empty write\nresolveEmptyTailRecords reads it as) and cut only an incomplete last\nrecord. A tail an interrupted append does not explain is left as it is\nand the integrity check quarantines the volume read-only as before.
Generated with [Devin](https://devin.ai)\n\nCo-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* rust volume: tests for the unindexed .dat tail repair
Reload volumes through the real load path after dropping .idx entries\nand cutting or damaging the .dat, mirroring Go's\nvolume_checking_tail_repair_test.go: complete unindexed needles are\nre-indexed (deletes as tombstones, empty writes as empty needles), an\nincomplete last record is cut, and a tail an interrupted append does\nnot explain is left untouched with the volume read-only.
Generated with [Devin](https://devin.ai)\n\nCo-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* volume: re-index a torn size 0 record as a live empty needle
A delete and a write with no data append the identical size 0 record, so a
recovered one cannot be told apart from the prior index state alone: the
write path accepts an empty overwrite of a live key and indexes it with
size 0, while a delete of the same key publishes a tombstone. Inferring a
tombstone from prior live state turned a lost index row after an empty
overwrite into a deletion -- the object read back not-found although the
write was answered.
Every re-indexed record is now indexed at its on-disk size, the entry
publishWriteRequest writes for an empty write. The other misread is
bounded: a recovered delete reads its key back as an empty object rather
than not-found, and the size 0 record carries no data to resurrect.
Review feedback on the Rust mirror (#11878): the same bug exists here;
fixing both sides identically.
Generated with [Devin](https://devin.ai)\n\nCo-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* rust volume: re-index a torn size 0 record as a live empty needle
Mirrors the Go fix in this pull request. A delete and a write with no data
append the identical size 0 record, so resolve_empty_tail_records could
not tell them apart from the prior index state: it treated a zero-size
record of a live id as a tombstone, although the write path indexes the
same bytes as a live size 0 entry. A crash after an empty overwrite of a
live key therefore recovered the object as deleted.
Every re-indexed record is now indexed at its on-disk size, the entry
publish_write writes for an empty write. A recovered delete reads its key
back as an empty object rather than not-found; the size 0 record carries
no data to resurrect.
Review: greptile-apps[bot] and devin-ai-integration[bot] on #11878.
Generated with [Devin](https://devin.ai)\n\nCo-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* volume: repair a .dat tail left past an empty .idx
The writable index loader creates the .idx before the first needle append,
so a crash after that append leaves a volume whose data nothing indexes.
CheckVolumeDataIntegrity skipped an empty index outright, so the volume
loaded writable and the surviving first needle stayed invisible while
later writes appended past it.
A zero-size index now runs the same repair with tailEntryPos -1: the walk
starts at the first record past the superblock with no anchor needle and
no append timestamp to compare against, gets the same complete-record
validation and incomplete-tail policy, and re-verifies through the normal
tail check once it wrote entries. A tail the repair cannot explain fails
the check, so the volume loads read-only instead of writable on an
unaccounted-for .dat.
Review feedback on the Rust mirror (#11878): the same bug exists here;
fixing both sides identically.
Generated with [Devin](https://devin.ai)\n\nCo-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* rust volume: repair a .dat tail left past an empty .idx
Mirrors the Go fix in this pull request. The writable index loader creates
the .idx before the first needle append, so a crash after that append
leaves a volume whose data nothing indexes. Both the repair and the
integrity check skipped an empty index, so the volume loaded writable and
the surviving first needle stayed invisible while later writes appended
past it.
An empty .idx now runs the same repair with no anchor: the walk starts at
the first record past the superblock, and a tail it cannot explain fails
the load like any other repair failure, since every later check skips an
empty index.
Review: devin-ai-integration[bot] on #11878.
Generated with [Devin](https://devin.ai)\n\nCo-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* volume: sync the .dat before publishing re-indexed .idx rows
A power loss after the repaired .idx rows become durable but before the
.dat tail records they name are flushed leaves the index pointing past
the end of the data file; the next load then keeps the volume read-only
until an operator steps in. Sync the .dat first so the rows never
outlive the records they reference.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* rust volume: sync the .dat before publishing re-indexed .idx rows
Mirrors the Go fix: a power loss after the repaired .idx rows become
durable but before the .dat tail records they name are flushed leaves
the index pointing past the end of the data file. Sync the .dat first.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* iceberg maintenance: compare parquet schemas by field id in bin-pack merge
parquet.EqualNodes ignores Iceberg field ids, so a column dropped and\nre-added under the same name and type compared equal and was merged,\nfiling the new column's values under the dropped field id. Require the\nfield ids to agree wherever both sides carry one; the v3 layout's\nper-field check uses the same test.\n\nGenerated with [Devin](https://devin.ai)\n\nCo-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* iceberg maintenance: merge bin files whose schemas differ by field id
A bin used to take the first file's Parquet schema as the writer's and\nrefuse every file that was not structurally identical, so a table never\ncompacted once files had a different column set or REQUIRED-vs-OPTIONAL\nrepetition. A file whose flat columns the writer's schema covers by\nfield id is converted into it (a column the file lacks reads as null);\na file that covers the writer's schema restarts the bin with it, so no\ncolumn is dropped. Equality-delete columns resolve on each file's own\nschema. Renames, type changes, nested columns and files without field\nids stay refused. The sorted and spatial merges are unchanged.\n\nGenerated with [Devin](https://devin.ai)\n\nCo-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* iceberg maintenance: test bin-pack merge across parquet schema spellings
Regression coverage for the field-id merge: an added column in either\norder, REQUIRED vs OPTIONAL, equality deletes resolved per file, the\nre-added column refused on v1/v2 and v3, and the refusals that stay\n(rename, type change, one-sided columns, an added required column, no\nfield ids).\n\nGenerated with [Devin](https://devin.ai)\n\nCo-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* iceberg maintenance: resolve missing delete columns as null in bin-pack merge
An equality delete on an optional column added after a data file was\nwritten no longer refuses the bin: the file reads the column as null,\nso a real delete key cannot match its rows and a null key deletes them.\n\nGenerated with [Devin](https://devin.ai)\n\nCo-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* iceberg maintenance: write bin-pack merges under the files' union schema
The writer schema is now the union of the bin's file schemas by field\nid, computed in a schema-only pass before any row is written: a column\nstays REQUIRED only where every file has it and spells it so, and the\nmerge refuses only true name/type/id conflicts. Files that need a union\nneither side covers (an added column plus a REQUIRED/OPTIONAL\ndifference, disjoint added columns) now merge in a single pass instead\nof refusing, and progressively wider bins no longer reread and rewrite\nevery earlier file per restart.\n\nGenerated with [Devin](https://devin.ai)\n\nCo-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* iceberg maintenance: spell the re-added column's test file for codespell
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* iceberg maintenance: resolve equality-delete columns by field id
A column renamed after its data and delete files were written still carries
its field id there; resolving it by id instead of the current name keeps a
rename from turning every equality key null. The missing-column null-key
fallback is now for data files only: an equality-delete file that lacks its
equality column is refused, as the spec requires it to carry them.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* iceberg maintenance: read only the parquet footer choosing a bin's writer schema
The union-schema pass downloaded every input in full, then the merge
downloaded each again. The footer suffices for the schema, and a chunked
file's ReadAt pulls only the chunks the footer spans.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* filer_pb: add MkdirExclusive
A taken name leaves the callback as a result, not an error, so filer
failover does not count a lost race as a filer failure.
* s3tables: create the entry with its attributes in one exclusive create
CreateTableBucket and CreateNamespace checked that the name was free and
then wrote the entry in several calls, so a second create in the window
replaced the first caller's entry including its owner. The create now
carries the marker, metadata and tags in one O_EXCL CreateEntry that
answers ErrEntryAlreadyExists with the same 409.
* s3: PutBucket uses an exclusive mkdir
The same window let a plain PutBucket replace a table bucket created
concurrently. The already-exists branch now answers from the entry itself,
since the registry just cached the name as absent, and the admin
auto-create path is exclusive too.
* s3: read the winning entry when a create loses the race
The concurrent-create branch called existingBucketError on a name the
registry had just cached as absent, answering NoSuchBucket, and the
auto-create fallback could assign an S3 owner to a table bucket it met
there. The negative cache is cleared before the existing-bucket error is
built, and a table bucket keeps its owner-less shape.
* s3: fail an auto-create that met a table bucket
A table bucket carries no S3 owner and must not gain one, and the upload
that auto-create ran for must not continue into it either.
* s3: fail an auto-create when the winning entry cannot be read
An unreadable existing entry fell through to success, so the write
continued into an unvalidated entry, possibly a table bucket.
* s3: resolve versionId tagging requests to the version-tagging actions
A request for an object's tags with versionId acted on a historical
version but authorized against the current-object tagging action, so an
explicit Deny on s3:*ObjectVersionTagging never applied.
* iam: map PutObjectVersionTagging and DeleteObjectVersionTagging to the tagging action
* s3: only resolve tagging when the query value is empty
The tagging routes match ?tagging with an empty value, so a non-empty one
is not a tagging request and would resolve a tagging grant onto the plain
read or version read that actually serves it.
* s3: keep resolving tagging for any query value
The routes match ?tagging whenever the key is present, so tagging=x is a
tagging request served by the tagging handlers and must resolve to the
tagging actions.
* s3tables: only the admin repairs a table bucket that has no owner metadata
CreateNamespace wrote bucket metadata naming the caller as owner before any
permission check ran, so an ordinary caller that found the marker without
metadata was written in as the bucket's owner and passed every check after.
A bucket in that state is answered with the operation's own denial; the
admin or a trusted internal handler may still repair it.
* s3tables: repair the marker-only bucket under the entry lock
A creator finishing its own create between the missing-metadata check and
the repair write would have its owner overwritten. The mutation now adopts
whatever metadata is there and writes only while it is still absent.
* s3tables: gate the repair on the Admin action, not an operation grant
hasAdminAction also answers s3tables:*, so a non-admin identity holding
the wildcard operation grant could pass the repair check and write itself
in as the bucket's owner.
* s3api: persist single-identity IAM changes through the credential manager
CreateAccessKey, DeleteAccessKey, UpdateAccessKey, PutUserPolicy, DeleteUserPolicy, SetUserStatus, UpdateUser and DeleteUser used to load the whole configuration, change one identity and then SaveConfiguration, rewriting and re-propagating every identity. They now write only the identity (and, for rename/delete, the groups and service accounts that referenced it) through the credential manager, which propagates that identity alone. Static identities and skipPersist keep the previous full-save/no-write contract.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* iamapi: persist single-identity IAM changes through the credential manager
Same targeted-write change as the embedded IAM API for the standalone weed iam dispatcher: UpdateUser, DeleteUser, CreateAccessKey, DeleteAccessKey, UpdateAccessKey, PutUserPolicy and DeleteUserPolicy write one identity through the credential manager instead of saving the whole configuration. CreateAccessKey keeps the full save when the handler created a user the store does not know.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* credential/memory: re-key identity on UpdateUser rename
UpdateUser stored the identity under the old username key even when identity.Name changed. Re-key it the way filer_etc does (delete old, save new) so the targeted rename path sees the moved record.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test: cover targeted single-identity IAM writes
Recording-store tests assert each rerouted action uses the single-identity credential store methods and never SaveConfiguration, in both the embedded S3 IAM dispatcher and the standalone iamapi dispatcher, plus the skipPersist and static-identity contracts and a memory-store rename case.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* credential: keep not-found and rename-collision semantics in stores
IamGrpcStore.DeleteAccessKey translates gRPC NotFound to ErrAccessKeyNotFound so a repeat delete stays the no-op the IAM handlers advertise. MemoryStore.UpdateUser rejects a rename onto an existing user instead of overwriting it, matching RenameUser.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* volume: repair an unindexed .dat tail at load instead of loading read-only
A needle write appends the needle to the .dat, then its entry to the .idx,
then answers the client. A crash between the two appends leaves a .dat that
ends past the last indexed needle; on the next start CheckVolumeDataIntegrity
reports "actual N bytes expected M bytes" and the volume loads read-only, and
a complete needle in that tail cannot be read because nothing indexes it.
After a process crash those bytes belong to a write that was never answered;
after a power loss a complete needle can survive in the .dat while its .idx
entry did not, although the client was answered. So complete needles are
re-indexed rather than cut.
CheckVolumeDataIntegrity now repairs the tail before the existing check,
which then runs on the repaired files as before. The repair:
- acts only on a local, version 3 volume within MaxPossibleVolumeSize, with a
whole number of .idx entries, that is about to load writable (neither
noWriteOrDelete nor noWriteCanDelete is set);
- requires the needle at the highest indexed offset to match its entry (id,
size, whole record, CRC);
- walks from the end of that needle to the real end of the file and gives
every complete needle (header, body, checksum, timestamp and padding inside
the file, CRC valid, append timestamp later than the needle before it) the
.idx entry the volume server would have written, in .dat order; a size 0
record, which is how a delete is appended, is indexed as a tombstone;
- cuts only an incomplete last record: fewer bytes than a header, a record
running past the end of the file, or a last record whose CRC fails;
- changes nothing otherwise (a damaged record with more data after it, a
header with id 0 or a negative size, append timestamps going backwards), so
the volume loads read-only exactly as before.
The .idx entries are written and synced before the .dat is truncated and
synced, so a crash during the repair leaves the needles indexed and the
remainder still past them; the next load cuts it. One warning per repaired
volume names the needles re-indexed and the bytes cut.
Tests: volume_checking_tail_repair_test.go reloads volumes through the normal
load path. Repaired: one and two complete unindexed needles, an unindexed
delete, a partial header, a partial body, a complete needle followed by a
partial one, a last needle with a bad CRC, a partial needle after an indexed
tombstone; each reloads clean afterwards. Left read-only and unchanged: the
last indexed needle with a wrong id or a bad CRC, a bad CRC followed by more
data, zero bytes past the last needle, an index past the data, a volume
persisted read-only, a partial .idx entry, and a clean volume.
TestCheckVolumeDataIntegrityVerifiesDeletionTail now also checks that on a
writable volume its 64 zero bytes are reported and kept.
Fixes#11864
* volume: fail the integrity check when a tail repair cannot complete
A repair that wrote the .idx or cut the .dat and then failed to sync only
logged the error; the check that followed saw the repaired files and the
volume loaded writable on a repair that may not be on disk. Any repair
error other than an unexplained tail now fails the check, so the volume
loads read-only, and the next load repairs again.
A failed .idx append could leave part of an entry behind, which made every
later load reject the .idx as a partial entry. The appended bytes are
truncated away again before the error is returned.
* volume: do not cut a tail record that could not be read
The walk cut a last record whose read failed for any reason, so an I/O
error (EIO) on a complete record erased it. Only a record whose bytes are
damaged (needle.ErrorCorrupted: a CRC mismatch or a body that does not
parse) is cut now; any other read error is returned with both files left
as they are, and the volume loads read-only.
* volume: treat a sized tail record with no data as damaged
A needle with no data is written with size 0. A tail record whose header
gives a body but whose data size field reads 0 parsed as an empty needle,
and ReadBytes checks the CRC only when there is data, so the repair indexed
it with its positive size and reads returned an empty body. Such a record
is now damaged like one with a bad CRC: cut when it is the last record,
otherwise the tail is left alone and the volume loads read-only.
* volume: keep an unindexed empty write as an empty needle
A delete and a write with no data both append a size 0 record, and the
repair indexed every size 0 record in the tail as a tombstone, so an
acknowledged empty write whose .idx entry was lost read back as deleted.
A delete is only appended for a needle that is live, so a size 0 record
of a needle that is absent or deleted (in the .idx and in the tail before
it) is an empty write and is re-indexed with size 0. One of a live needle
stays a tombstone, the reading weed fix and the backup's index
regeneration already give it. The .idx is scanned only when the tail holds
a size 0 record.
* volume: cut torn-tail test records inside their body, not at +40
The partial-record cases cut the .dat 40 bytes into a record. A needle
with up to 7 bytes of data is a 40-byte record, so newRandomNeedle's
random length (1..1023) left it whole about once in 150 runs per case:
the repair then correctly re-indexed it and cut nothing, and the test
failed on the .dat size. The cut now lands halfway through the record's
body, which is always past its header and short of its end.
---------
Co-authored-by: Javier de la Torre <jatorre@lakehousebox.com>
* volume: run the durable sync outside the reader-visible lock
Reads of a volume waited on that volume's fsync because the write lock
was held across Sync()/sync_all. A new per-volume lock (Go
dataFileSyncLock, Rust data_file_sync_lock) now serializes every .dat
mutation and lifecycle op across append+publish+sync+rollback, while the
fsync itself runs with dataFileAccessLock/the io write guard dropped —
so readers proceed during a sync, and a failed sync's rollback still
cannot clobber another write or a compaction swap.
Same change in Go (volume_write.go syncWrite/processBatch, SyncToDisk)
and Rust (write_needles_grouped/write_synced_run).
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* volume: publish after sync; fix Close/CommitCompact lock order
Address review on #11863:
- Close took dataFileSyncLock before waiting on isCompactionInProgress
while CommitCompact claims the flag then takes the lock — a deadlock.
Close now claims the flag before taking the locks (flag -> sync -> access).
- Go now publishes needle-map entries only after the .dat sync, matching
Rust's ordering: appendWriteRequest/appendDeleteRequest stage the bytes,
publishWriteRequest/publishDeleteRequest index them after Sync. A failed
sync rolls back bytes no reader ever resolved — the streamed-read
offset-invalidation window is gone. processBatch splits runs at repeated
needle ids (Rust parity) so dedup/cookie checks stay sequential; the old
snapshot/restore rollback machinery is removed.
- Rust commit_compact (store-lock caller) takes data_file_sync_lock via
try_lock -> WouldBlock before the write lease; commit_compact_held and
the tier-down _held paths document/require the caller-held sync lock,
and the two vol.instance().write_lock() grpc sites take it first. The
swap can no longer land between a durable write's append and publish.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* volume: restore write watermarks when an inline fsync fails
The syncWrite rollback truncated the .dat but left lastAppendAtNs,
lastWriteAppendAtNs, lastWriteNeedleKey, lastWriteDeleted and
lastModifiedTsSeconds describing the rejected write; recovery and
compaction could read a watermark for a record that no longer exists.
Snapshot and restore them on failure, as processBatchRun already does.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* s3: mask anonymous misses behind AccessDenied unless the bucket is listable
Deferred anonymous GET/HEAD authorization only runs once an entry is
selected, so misses returned 404 while existing keys returned 403 — an
unauthenticated existence oracle for bucket names and object keys.
Route not-found responses in Get/HeadObjectHandler through a check that
only discloses 404 when anonymous listing is allowed (policy, identity or
a public-read bucket); otherwise the miss is indistinguishable from a
denied read, matching AWS and 4.48 behavior.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* s3: mask missing-directory misses for anonymous reads too
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* s3: check a fixed request shape when masking anonymous misses
The miss check passed the caller's request through, so query parameters
could retarget the resolved action (?policy) or satisfy an s3:prefix
condition on an unrelated subtree. Each check now runs on a request with a
fixed shape: a ListBucket scoped to the probed key and a plain read of the
key, so the miss only shows when the caller could learn the same fact
directly.
* s3: mask conditional-header misses for anonymous reads
An If-Match conditional on a missing key wrote NoSuchKey directly from
processConditionalHeaders, before the handlers reached the masked miss
path, so anonymous callers could still distinguish a missing key from a
denied read.
* s3: keep the versionId when masking anonymous misses
The read leg must evaluate the action the caller's request resolves to:
a versioned miss is s3:GetObjectVersion, and dropping versionId would let a
plain GetObject grant disclose versions the caller may not read.
---------
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
The auto-select branches of ReceiveFile (is_ec_volume) and
VolumeEcShardsCopy passed the build-default DATA_SHARDS_COUNT into
find_ec_shard_target_location, so its free-slot maths scored every
volume as 10+4 even when the volume's .vif declares a custom ratio
(e.g. 6+2). A wider-ratio volume's shards could be steered onto a disk
too full to hold them all, and a narrower one would be refused slots
it actually has.
Add Store::lookup_ec_data_shards(collection, vid), which answers from
the mounted EC volume first and then from the .vif on whichever disk
holds it (via read_ec_shard_config_across_dirs, so split -dir/-dir.idx
and rebuild-staging layouts are covered), falling back to the default
when no metadata exists yet.
Mirrors the enterprise fix (seaweed-mono c22fa8a15).
* master: detect and quarantine duplicate volume ids across layouts (#11763)
* shell: volume.check.duplicate_ids audit command (#11763)
* master: persist duplicate-vid quarantine across heartbeats (#11763)
* master: quarantine all copies of a duplicated volume id
Near-simultaneous registrations let each side quarantine only the other,
so arrival order decided which copy stayed writable. A confirmed
cross-layout collision means either copy could be the impostor, so flag
every colliding copy read-only regardless of registration order.
* topology: return a copy from VolumeLayout.Lookup
Callers iterating the returned slice raced with registrations and
removals replacing the location list in place.
* master: release duplicate-vid quarantine when the conflict resolves
A cross-node volume.move briefly shows the same id on two nodes under
different disk layouts; permanent quarantine would pin a valid target
read-only. Keep the first-reported state per quarantined copy and lift
the marks once the conflicting copy unregisters, so only a persistent
collision stays read-only.
* shell: include ec shards in volume.check.duplicate_ids
Ec-only volumes share the global id space. Compare shard collections
against regular ones while allowing shards and regular copies of the
same encoded volume.
* master: keep quarantine release current and cover all removal paths
Track the latest server-reported read-only flag while quarantined so a
release restores the server s true state, and run the release check on
collection deletion and data-node removal too, not only explicit volume
unregistration.
* shell: do not flag ec-only volumes as duplicates
An ec-only volume has no regular copy, so its shard collections had
nothing to compare against and always read as foreign.
* master: quarantine every conflicting layout at the layout level (#11763)
- collect all conflicting layouts, not just the first, so a vid spanning
three layouts does not leave one writable
- mark quarantine on the location list only: folding it into the stored
report made the volume digest permanently disagree with the server and
forced a full volume list every heartbeat
- isAllWritable now honors the location read-only mark so a quarantined
copy stays out of writables through resyncs and re-registration
- re-check the conflict before keeping marks, and do not clear a node's
quarantine while it still holds the vid in another layout
- log the full layout keys on both sides of the conflict
* master: push duplicate-vid quarantine to the volume server (#11763)
A quarantined copy stayed writable on the volume server itself for a
client holding an existing file id, since the master-only mark never
reached the server's local write path. Quarantine now also calls
VolumeMarkReadonly (can_delete so expiry still drains), is re-pushed if
the server comes back reporting it writable, and VolumeMarkWritable
runs when the conflict resolves and the server had reported writable.
* master: retry unenforced quarantine marks and restore server writability on release (#11763)
Dial the advertised grpc port, return mark errors, and keep a
quarantine entry until the writable push succeeds so a released
copy does not stay read-only on the server.
* master: treat rp-only layout differences as an in-flight replication change (#11763)
volume.configure.replication rolls through nodes one at a time, so a
volume legitimately reports the old and new replica placement while it
settles. Layouts differing only in rp within the same collection are a
supported transition, not a reused id.
* shell: group ec shards by collection in the duplicate id audit (#11763)
EC shards of one volume legitimately span disk types when
DiskTypePrefer spills, so grouping by disk type reports healthy
volumes as duplicates.
2026-10-20 07:37:33 -07:00
1831 changed files with 230776 additions and 18372 deletions
@@ -17,7 +17,9 @@ SeaweedFS is a simple and highly scalable distributed file system. There are two
1. to store billions of files!
2. to serve the files fast!
One `weed` binary serves an S3 object store, a POSIX file system, and a lakehouse with S3 Tables, all over the same data. Each blob is one disk read away, capacity grows by starting another volume server, and cloud storage can be cached or tiered transparently. Both read and write operations have O(1) complexity and can run at the full speed supported by the underlying hardware.
One SeaweedFS cluster serves an S3 object store, a POSIX file system, a lakehouse with S3 Tables and a built-in Iceberg REST Catalog, and [vector search with S3 Vector Buckets](#vector-search-with-s3-vector-buckets), all on shared storage. Each blob is one disk read away, capacity grows by starting another volume server, and cloud storage can be cached or tiered transparently. Blob reads and writes have O(1) complexity and can run at the full speed supported by the underlying hardware.

- [Download Binaries for different platforms](https://github.com/seaweedfs/seaweedfs/releases/latest)
For SeaweedFS Enterprise binaries, pass `--enterprise` to the script: `... | bash -s -- --enterprise`.
Then start a ready-to-use S3 object store:
```bash
@@ -204,12 +209,13 @@ On one laptop, [`weed benchmark`][Benchmarks] writes 1KB files at 15,700 per sec
## The most complete S3 API ##
The S3 gateway implements the object, bucket, S3 Tables, IAM, and STS APIs on one endpoint, so the AWS SDKs and CLI, rclone, restic, Spark, and Trino work unchanged.
The S3 gateway implements the object, bucket, S3 Tables, S3 Vectors, IAM, and STS APIs on one endpoint, so the AWS SDKs and CLI, rclone, restic, Spark, and Trino work unchanged. S3 Vectors uses the `weed-vector` service described [below](#vector-search-with-s3-vector-buckets).
| API | Operations |
| --- | --- |
| S3 bucket and object | 73 |
| S3 Tables | 36 |
| S3 Vectors | 19 |
| IAM | 39 |
| STS | 5 |
@@ -223,15 +229,49 @@ The full operation list is in [Amazon S3 API][AmazonS3API], and [Supported APIs
## A data warehouse with S3 Tables ##
SeaweedFS is a lakehouse in one system. [S3 Table Buckets][S3TableBucket] hold Apache Iceberg tables by default, or [Lance][LanceCatalog] tables for vectors and multimodal data, and the built-in [Iceberg REST Catalog][IcebergCatalog] and Lance namespace serve them directly. There is no Hive Metastore, Glue, or separate catalog service to deploy, secure, and back up.
SeaweedFS is a lakehouse in one system. [S3 Table Buckets][S3TableBucket] hold Apache Iceberg tables by default, or [Lance][LanceCatalog] tables for vectors and multimodal data, and the built-in [Iceberg REST Catalog][IcebergCatalog] and Lance namespace serve them directly. The S3 Tables API and Iceberg REST Catalog share the same namespaces, table metadata, and permissions. There is no Hive Metastore, Glue, or separate catalog service to deploy, secure, and back up.
* Query engines operate on the same tables at the same time: [Spark][SparkIceberg], [Trino][TrinoIceberg], [Dremio][DremioIceberg], [DuckDB][DuckDBIceberg], [Apache Doris][DorisIceberg], [RisingWave][RisingWaveIceberg], ClickHouse, and [LanceDB][LanceDB]. Catalog commits are atomic compare-and-swap, so concurrent writers are safe. [Lakekeeper][Lakekeeper] can front the same storage with STS-vended credentials. [OLake][OLake] replicates PostgreSQL, MySQL, and MongoDB into the same tables with change data capture.
* Query engines operate on the same tables at the same time: [Spark][SparkIceberg], [Trino][TrinoIceberg], [Dremio][DremioIceberg], [DuckDB][DuckDBIceberg], [Apache Doris][DorisIceberg], [RisingWave][RisingWaveIceberg], ClickHouse, and [LanceDB][LanceDB]. Catalog commits are atomic compare-and-swap, so concurrent writers are safe. [Lakekeeper][Lakekeeper] can front the same storage with STS-vended credentials. [OLake Go][OLake] is a high-performance, open-source EL (Extract–Load) platform that bridges operational databases and open lakehouse storage - writing Apache Iceberg tables or Parquet files on Amazon S3.
* [Automated table maintenance][IcebergMaintenance]: compaction, snapshot expiration, orphan file removal, and manifest rewriting, configured per bucket or table through the S3 Tables maintenance APIs, and the same for [Lance][LanceMaintenance].
* IAM at the bucket, namespace, and table level with standard bucket policies, see [S3 Tables Security][S3TablesSecurity].
* A [Hadoop compatible file system][Hadoop] for Spark, Flink, and HBase.
`S3_TABLE_BUCKET=warehouse ./weed mini -dir=./data` brings the whole stack up on a laptop.
## Vector search with S3 Vector Buckets ##
Keep embeddings alongside your objects and tables in the same SeaweedFS cluster. S3 Vector Buckets expose the AWS S3 Vectors API for semantic search, retrieval-augmented generation (RAG), and recommendations, using Lance tables on SeaweedFS storage.
* Create vector buckets and indexes; insert, retrieve, list, and delete vectors with the AWS SDKs or CLI pointed at your S3 endpoint.
* Find the nearest vectors with cosine or Euclidean distance, filter by metadata, and return distances and metadata with the results.
* Control access with `s3vectors:*` IAM actions and vector bucket policies; tag buckets and indexes for organization and access control.
Install the separate vector engine; the default installer installs only `weed`:
Release 4.53 provides vector binaries for Linux amd64 and arm64. If your release has no binary for your platform, build from [source](seaweed-vector). From a directory where you want a new checkout:
If you already have a checkout, start at its root and skip the clone and `cd seaweedfs` steps.
For Windows source builds, set `PROTOC` to a native `protoc.exe` before running Cargo; the [desktop build workflow](.github/workflows/rust-vector-platforms.yml) shows how to select the vendored executable.
Run the installed `weed-vector` binary alongside the local cluster started with `weed mini` above (or use `seaweed-vector/target/release/weed-vector` after building from source):
```bash
weed-vector --masters=localhost:9333
```
It registers with the master and the S3 gateway discovers it automatically. An AWS S3 Vectors client uses the same S3 endpoint at http://localhost:8333. The vector engine stores its data through the cluster's filers and volume servers, so embeddings share the storage layer's replication and erasure coding. See the [AWS SDK examples](test/s3vectors/s3vectors_test.py) for bucket, index, and query operations.
## A fast cache for cloud storage ##
[Cloud Drive][CloudDrive] mounts a bucket from S3, Google Cloud Storage, Azure, Backblaze B2, Wasabi, Storj, or any S3-compatible store into SeaweedFS and serves it at local speed:
@@ -487,7 +527,7 @@ Your support will be really appreciated by me and other supporters!
* master/filer/volume are stateful sets with anti-affinity on the hostname,
so your deployment will be spread/HA.
* leveldb2 is the default filer backend; a mysql-compatible database (memsql, ...) enables HA (multiple filer instances) and the backup/HA it can provide.
* with `filer.extraEnvironmentVars.WEED_MYSQL_ENABLED` set to `"true"`, mysql user/password are created in a k8s secret (default: `<release>-seaweedfs-db-secret`) and injected to the filer with ENV. On any other store neither the secret nor the `WEED_MYSQL_*` env, plain or secret-backed, is rendered.
* with `filer.extraEnvironmentVars.WEED_MYSQL_ENABLED` set to `"true"`, mysql user/password are created in a k8s secret (default: `<release>-seaweedfs-db-secret`) and injected to the filer with ENV. The secret is kept on `helm uninstall`, and a later install reuses the user and password in it, including ones you edited in. On any other store neither the secret nor the `WEED_MYSQL_*` env, plain or secret-backed, is rendered.
* cert config exists and can be enabled, but not been tested, requires cert-manager to be installed.
`namespaceOverride` deploys every namespaced object into an existing namespace other than the release namespace, for instance when the chart is a subchart. In-cluster addresses follow it. Set it before the first install: changing it later moves regular namespaced objects only — claims and their data do not move, hook resources such as the resize hook's ServiceAccount, Role, RoleBinding, and NetworkPolicy stay behind in the old namespace, and generated Secrets are not recreated in the new one. Migrate claims by hand and clean up the old hook resources. If the release uses generated S3 credentials, copy `<fullname>-s3-secret`, including its key fields, into the new namespace before the upgrade — otherwise the chart generates new keys and clients on the old ones fail authentication (explicit `s3.credentials` and `credentials.*.existingSecret` values do not use this generation path). With the default MySQL setup, also copy `<fullname>-db-secret` — a `pre-install` hook only, so it never reaches the new namespace — or pass `WEED_MYSQL_USERNAME` and `WEED_MYSQL_PASSWORD` through `filer.extraEnvironmentVars`. With the default `hostPath` storage the data is node-local too: stop the old workloads and plan the hostPath migration first, or the new pods can come up empty on other nodes or share the same directories concurrently.
## Gateway API
Every component with an `ingress` block (master, volume, filer, s3 and admin) can also be exposed through a [Gateway API](https://gateway-api.sigs.k8s.io/) `HTTPRoute`, for clusters that route through a Gateway instead of an Ingress controller. `<component>.httpRoute` sits next to `<component>.ingress` and is disabled by default. The chart does not create the Gateway, so point `parentRefs` at one that already exists, and the `gateway.networking.k8s.io/v1` CRDs must be installed.
@@ -565,6 +569,8 @@ s3:
With `rules` left empty the route sends all traffic to the component's own Service and port. A rule may set `matches`, `filters`, `timeouts` and `backendRefs`, and a rule without `backendRefs` still routes to that Service. In all-in-one mode the master, volume, filer and s3 routes target the all-in-one Service. When S3 runs only on the filer (`filer.s3.enabled` without `s3.enabled`), the s3 route uses `filer.s3.port`.
Outside all-in-one mode, the volume Ingress and the default volume route target the `client-volume` Service, which lists only Ready volume pods. The `volume` Service also publishes pods that are not Ready yet, so their DNS records can exist before readiness. An explicit `backendRefs` entry that names the `volume` Service can therefore reach a volume server before it is Ready. Name the `client-volume` Service instead, with `volume.port`, and copy its `metadata.name` from `helm template` because long release names are truncated.
## Network Policies
In a namespace with a default-deny policy the install hangs: the components cannot resolve each other, and the post-install bucket hook waits on the master and filer until it gives up. `networkPolicy.enabled` renders one `NetworkPolicy` per component, selecting its pods by the standard `app.kubernetes.io/{name,instance,component}` labels and admitting traffic from the other pods of the release on the ports that component listens on.
@@ -606,6 +612,21 @@ Two things worth knowing before you turn this on:
The DNS selectors default to CoreDNS as kubeadm, kind and the managed offerings from AWS, Google and Azure install it. On OpenShift, override `egress.dnsNamespaceSelector` and `egress.dnsPodSelector` to match `openshift-dns`; see the comment in `values.yaml`.
## Volume server HTTPS
With `global.seaweedfs.enableSecurity`, gRPC between components uses mutual TLS, but the volume servers' HTTP port, which carries the file data, stays plaintext. Set `global.seaweedfs.volumeHttps.enabled: true` to serve it over TLS:
- `[https.volume]` points at the volume certificate, which then also names every volume group's service (`*.<fullname>-volume.<namespace>`), since volume servers register as `<pod>.<fullname>-volume.<namespace>`. With `certificates.externalCertificates`, the provided `<fullname>-volume-cert` needs those names.
- `[https.client]` is enabled with the client certificate and the chart CA, so every chart pod that talks to the volume servers does it over HTTPS.
- The volume probes use HTTPS.
- Every pod that reads `security.toml` gets a `checksum/volume-https` annotation, so switching the option on or off restarts them together.
The volume servers do not require client certificates, because kubelet probes cannot present one; enable JWT signing (`securityConfig.jwtSigning`) to authorize volume reads and writes. `[https.client]` also sends the admin UI's HTTP lookups on masters and S3 over HTTPS, which their plain listeners refuse, so the UI can miss volume public URLs and refuse SSE-S3 vector buckets. A volume Ingress or HTTPRoute needs its controller's HTTPS-backend setting. `weed mount`, the CSI driver and other clients outside the chart need their own `[https.client]` with the chart CA. The option is not supported with `allInOne`, whose volume server registers by pod IP.
## Common labels
`commonLabels` adds labels to the metadata of every object the chart creates. It may be a map or a template string rendering to one. The chart's own labels cannot be overridden, and pods (see `podLabels`) and `volumeClaimTemplates` are left unchanged.
## Pod and container security contexts
Pod and container security contexts are configurable independently for every built-in workload and remain empty by default for backwards compatibility. The examples in `values.yaml` show how to enable a `RuntimeDefault` seccomp profile, disable privilege escalation and privileged mode, drop all Linux capabilities, and use a read-only root filesystem.
@@ -679,3 +700,47 @@ generates a new UUID and the license stops matching. See
[Master data](#master-data-hostpath-vs-a-claim). Check the binding with
The S3 gateway serves the S3 Vectors API on its existing endpoint. Enable a vector engine and Lance maintenance worker alongside the cluster:
```yaml
s3:
enabled: true
vector:
enabled: true
replicas: 1
worker:
enabled: true
admin:
enabled: true
secret:
adminPassword: change-me
global:
seaweedfs:
securityConfig:
jwtSigning:
filerWrite: true
```
The engine registers its pod address with the masters and stores indexes through the filers and volume servers. Both Go and Rust volume servers use the same setup. The maintenance worker builds and updates ANN indexes; queries can scan data before indexing completes. Enabling filer write signing also requires clients writing directly to the filer HTTP API to supply a signed JWT.
The bundled engine images support Linux amd64 and arm64. Use `vector.imageOverride` to select a compatible engine build, `vector.masters` for an external cluster, and `vector.resources` to set memory and CPU limits. The cache defaults to 512 MiB and is configured with `vector.readCacheMB`.
`global.seaweedfs.enableSecurity` provisions and mounts the vector certificate and shared security configuration. With external certificates, provide the `<fullname>-vector-cert` secret (`my-seaweedfs-vector-cert` for release `my`). `fullname` follows `seaweedfs.fullname`, including `fullnameOverride`, `nameOverride`, and DNS-name truncation. Provide this secret alongside the existing CA and client secrets.
The default `vector.metricsPort: 0` keeps TCP probes. Set `vector.metricsPort: 9329` to serve `/metrics`, `/health`, and `/ready` from the engine; readiness then checks master registration. Images older than 4.54 do not accept the metrics flag, so leave metrics off when pinning `image.tag` or `vector.imageOverride` to one. Enabling `global.seaweedfs.monitoring.enabled` also creates the vector ServiceMonitor. NetworkPolicy includes the configured gRPC and metrics ports.
To provision vector buckets during installation and upgrades, set:
```yaml
vector:
enabled: true
createBuckets:
- embeddings
```
The hook uses the AWS CLI's S3 Vectors API and works with existing SeaweedFS vector releases. It creates missing buckets, preserves existing vector buckets, and fails if the name belongs to an Iceberg bucket. It uses the chart's generated S3 admin credentials, including `s3.credentials.admin.existingSecret`. With a custom `s3.existingConfigSecret` or an external endpoint, set `vector.bucketHook.credentialsSecret` to a Secret containing `AWS_ACCESS_KEY_ID` and `AWS_SECRET_ACCESS_KEY` (and optionally `AWS_SESSION_TOKEN`). Key names are configurable. `vector.bucketHook.endpoint` overrides the internal S3 service URL; `caBundleSecret` and `caBundleKey` configure a private HTTPS CA. The hook image and resource limits are configurable under `vector.bucketHook`. For restricted Pod Security, select a non-root-compatible image and configure `vector.bucketHook.podSecurityContext` and `containerSecurityContext`, including `runAsNonRoot` and the image user ID.
"description":"Vector engine metrics require HTTP metrics to be enabled. Gateway bucket labels require vector bucket attribution.",
"fieldConfig":{
"defaults":{
"color":{
"mode":"palette-classic"
},
"custom":{
"axisBorderShow":false,
"axisCenteredZero":false,
"axisColorMode":"text",
"axisLabel":"",
"axisPlacement":"auto",
"barAlignment":0,
"drawStyle":"line",
"fillOpacity":0,
"gradientMode":"none",
"hideFrom":{
"legend":false,
"tooltip":false,
"viz":false
},
"insertNulls":false,
"lineInterpolation":"linear",
"lineWidth":1,
"pointSize":4,
"scaleDistribution":{
"type":"linear"
},
"showPoints":"auto",
"spanNulls":false,
"stacking":{
"group":"A",
"mode":"none"
},
"thresholdsStyle":{
"mode":"off"
}
},
"mappings":[],
"thresholds":{
"mode":"absolute",
"steps":[
{
"color":"green",
"value":null
}
]
},
"unit":"reqps",
"unitScale":true
},
"overrides":[]
},
"gridPos":{
"h":8,
"w":12,
"x":0,
"y":140
},
"id":96,
"links":[],
"options":{
"legend":{
"calcs":[],
"displayMode":"list",
"placement":"bottom",
"showLegend":true
},
"tooltip":{
"mode":"single",
"sort":"none"
}
},
"pluginVersion":"8.1.2",
"targets":[
{
"datasource":{
"type":"prometheus",
"uid":"${DS_PROMETHEUS}"
},
"editorMode":"code",
"expr":"sum by (type, bucket) (rate(SeaweedFS_s3_request_total{namespace=~\"$NAMESPACE\",type=~\"s3vectors\\\\..*\"}[$__rate_interval]))",
"range":true,
"refId":"A",
"legendFormat":"{{type}} {{bucket}}"
}
],
"title":"Vector API requests by bucket",
"type":"timeseries"
},
{
"datasource":{
"type":"prometheus",
"uid":"${DS_PROMETHEUS}"
},
"description":"Vector engine metrics require HTTP metrics to be enabled. Gateway bucket labels require vector bucket attribution.",
"fieldConfig":{
"defaults":{
"color":{
"mode":"palette-classic"
},
"custom":{
"axisBorderShow":false,
"axisCenteredZero":false,
"axisColorMode":"text",
"axisLabel":"",
"axisPlacement":"auto",
"barAlignment":0,
"drawStyle":"line",
"fillOpacity":0,
"gradientMode":"none",
"hideFrom":{
"legend":false,
"tooltip":false,
"viz":false
},
"insertNulls":false,
"lineInterpolation":"linear",
"lineWidth":1,
"pointSize":4,
"scaleDistribution":{
"type":"linear"
},
"showPoints":"auto",
"spanNulls":false,
"stacking":{
"group":"A",
"mode":"none"
},
"thresholdsStyle":{
"mode":"off"
}
},
"mappings":[],
"thresholds":{
"mode":"absolute",
"steps":[
{
"color":"green",
"value":null
}
]
},
"unit":"s",
"unitScale":true
},
"overrides":[]
},
"gridPos":{
"h":8,
"w":12,
"x":12,
"y":140
},
"id":97,
"links":[],
"options":{
"legend":{
"calcs":[],
"displayMode":"list",
"placement":"bottom",
"showLegend":true
},
"tooltip":{
"mode":"single",
"sort":"none"
}
},
"pluginVersion":"8.1.2",
"targets":[
{
"datasource":{
"type":"prometheus",
"uid":"${DS_PROMETHEUS}"
},
"editorMode":"code",
"expr":"histogram_quantile(0.95, sum by (le, bucket) (rate(SeaweedFS_s3_request_seconds_bucket{namespace=~\"$NAMESPACE\",type=\"s3vectors.QueryVectors\"}[$__rate_interval])))",
"range":true,
"refId":"A",
"legendFormat":"{{bucket}}"
}
],
"title":"Vector query p95 latency",
"type":"timeseries"
},
{
"datasource":{
"type":"prometheus",
"uid":"${DS_PROMETHEUS}"
},
"description":"Vector engine metrics require HTTP metrics to be enabled. Gateway bucket labels require vector bucket attribution.",
"fieldConfig":{
"defaults":{
"color":{
"mode":"palette-classic"
},
"custom":{
"axisBorderShow":false,
"axisCenteredZero":false,
"axisColorMode":"text",
"axisLabel":"",
"axisPlacement":"auto",
"barAlignment":0,
"drawStyle":"line",
"fillOpacity":0,
"gradientMode":"none",
"hideFrom":{
"legend":false,
"tooltip":false,
"viz":false
},
"insertNulls":false,
"lineInterpolation":"linear",
"lineWidth":1,
"pointSize":4,
"scaleDistribution":{
"type":"linear"
},
"showPoints":"auto",
"spanNulls":false,
"stacking":{
"group":"A",
"mode":"none"
},
"thresholdsStyle":{
"mode":"off"
}
},
"mappings":[],
"thresholds":{
"mode":"absolute",
"steps":[
{
"color":"green",
"value":null
}
]
},
"unit":"reqps",
"unitScale":true
},
"overrides":[]
},
"gridPos":{
"h":8,
"w":12,
"x":0,
"y":148
},
"id":98,
"links":[],
"options":{
"legend":{
"calcs":[],
"displayMode":"list",
"placement":"bottom",
"showLegend":true
},
"tooltip":{
"mode":"single",
"sort":"none"
}
},
"pluginVersion":"8.1.2",
"targets":[
{
"datasource":{
"type":"prometheus",
"uid":"${DS_PROMETHEUS}"
},
"editorMode":"code",
"expr":"sum by (type, code) (rate(SeaweedFS_s3_request_total{namespace=~\"$NAMESPACE\",type=~\"s3vectors\\\\..*\",code=~\"[45]..\"}[$__rate_interval]))",
"range":true,
"refId":"A",
"legendFormat":"{{type}} {{code}}"
}
],
"title":"Vector API errors",
"type":"timeseries"
},
{
"datasource":{
"type":"prometheus",
"uid":"${DS_PROMETHEUS}"
},
"description":"Vector engine metrics require HTTP metrics to be enabled. Gateway bucket labels require vector bucket attribution.",
"fieldConfig":{
"defaults":{
"color":{
"mode":"palette-classic"
},
"custom":{
"axisBorderShow":false,
"axisCenteredZero":false,
"axisColorMode":"text",
"axisLabel":"",
"axisPlacement":"auto",
"barAlignment":0,
"drawStyle":"line",
"fillOpacity":0,
"gradientMode":"none",
"hideFrom":{
"legend":false,
"tooltip":false,
"viz":false
},
"insertNulls":false,
"lineInterpolation":"linear",
"lineWidth":1,
"pointSize":4,
"scaleDistribution":{
"type":"linear"
},
"showPoints":"auto",
"spanNulls":false,
"stacking":{
"group":"A",
"mode":"none"
},
"thresholdsStyle":{
"mode":"off"
}
},
"mappings":[],
"thresholds":{
"mode":"absolute",
"steps":[
{
"color":"green",
"value":null
}
]
},
"unit":"short",
"unitScale":true
},
"overrides":[]
},
"gridPos":{
"h":8,
"w":12,
"x":12,
"y":148
},
"id":99,
"links":[],
"options":{
"legend":{
"calcs":[],
"displayMode":"list",
"placement":"bottom",
"showLegend":true
},
"tooltip":{
"mode":"single",
"sort":"none"
}
},
"pluginVersion":"8.1.2",
"targets":[
{
"datasource":{
"type":"prometheus",
"uid":"${DS_PROMETHEUS}"
},
"editorMode":"code",
"expr":"last_over_time(SeaweedFS_vector_connected{namespace=~\"$NAMESPACE\"}[5m]) * on (cluster, namespace, instance, job) up{namespace=~\"$NAMESPACE\"}",
"range":true,
"refId":"A",
"legendFormat":"{{instance}}"
}
],
"title":"Registered vector engines",
"type":"timeseries"
},
{
"datasource":{
"type":"prometheus",
"uid":"${DS_PROMETHEUS}"
},
"description":"Vector engine metrics require HTTP metrics to be enabled. Gateway bucket labels require vector bucket attribution.",
"fieldConfig":{
"defaults":{
"color":{
"mode":"palette-classic"
},
"custom":{
"axisBorderShow":false,
"axisCenteredZero":false,
"axisColorMode":"text",
"axisLabel":"",
"axisPlacement":"auto",
"barAlignment":0,
"drawStyle":"line",
"fillOpacity":0,
"gradientMode":"none",
"hideFrom":{
"legend":false,
"tooltip":false,
"viz":false
},
"insertNulls":false,
"lineInterpolation":"linear",
"lineWidth":1,
"pointSize":4,
"scaleDistribution":{
"type":"linear"
},
"showPoints":"auto",
"spanNulls":false,
"stacking":{
"group":"A",
"mode":"none"
},
"thresholdsStyle":{
"mode":"off"
}
},
"mappings":[],
"thresholds":{
"mode":"absolute",
"steps":[
{
"color":"green",
"value":null
}
]
},
"unit":"reqps",
"unitScale":true
},
"overrides":[]
},
"gridPos":{
"h":8,
"w":12,
"x":0,
"y":156
},
"id":100,
"links":[],
"options":{
"legend":{
"calcs":[],
"displayMode":"list",
"placement":"bottom",
"showLegend":true
},
"tooltip":{
"mode":"single",
"sort":"none"
}
},
"pluginVersion":"8.1.2",
"targets":[
{
"datasource":{
"type":"prometheus",
"uid":"${DS_PROMETHEUS}"
},
"editorMode":"code",
"expr":"sum by (type, code) (rate(SeaweedFS_vector_request_total{namespace=~\"$NAMESPACE\",code!=\"OK\"}[$__rate_interval]))",
"range":true,
"refId":"A",
"legendFormat":"{{type}} {{code}}"
}
],
"title":"Vector engine request failures",
"type":"timeseries"
},
{
"datasource":{
"type":"prometheus",
"uid":"${DS_PROMETHEUS}"
},
"description":"Vector engine metrics require HTTP metrics to be enabled. Gateway bucket labels require vector bucket attribution.",
"fieldConfig":{
"defaults":{
"color":{
"mode":"palette-classic"
},
"custom":{
"axisBorderShow":false,
"axisCenteredZero":false,
"axisColorMode":"text",
"axisLabel":"",
"axisPlacement":"auto",
"barAlignment":0,
"drawStyle":"line",
"fillOpacity":0,
"gradientMode":"none",
"hideFrom":{
"legend":false,
"tooltip":false,
"viz":false
},
"insertNulls":false,
"lineInterpolation":"linear",
"lineWidth":1,
"pointSize":4,
"scaleDistribution":{
"type":"linear"
},
"showPoints":"auto",
"spanNulls":false,
"stacking":{
"group":"A",
"mode":"none"
},
"thresholdsStyle":{
"mode":"off"
}
},
"mappings":[],
"thresholds":{
"mode":"absolute",
"steps":[
{
"color":"green",
"value":null
}
]
},
"unit":"short",
"unitScale":true
},
"overrides":[]
},
"gridPos":{
"h":8,
"w":12,
"x":12,
"y":156
},
"id":101,
"links":[],
"options":{
"legend":{
"calcs":[],
"displayMode":"list",
"placement":"bottom",
"showLegend":true
},
"tooltip":{
"mode":"single",
"sort":"none"
}
},
"pluginVersion":"8.1.2",
"targets":[
{
"datasource":{
"type":"prometheus",
"uid":"${DS_PROMETHEUS}"
},
"editorMode":"code",
"expr":"sum by (type) (SeaweedFS_vector_in_flight_requests{namespace=~\"$NAMESPACE\"})",
"range":true,
"refId":"A",
"legendFormat":"{{type}}"
}
],
"title":"Vector engine requests in flight",
"type":"timeseries"
},
{
"datasource":{
"type":"prometheus",
"uid":"${DS_PROMETHEUS}"
},
"description":"Vector engine metrics require HTTP metrics to be enabled. Gateway bucket labels require vector bucket attribution.",
"fieldConfig":{
"defaults":{
"color":{
"mode":"palette-classic"
},
"custom":{
"axisBorderShow":false,
"axisCenteredZero":false,
"axisColorMode":"text",
"axisLabel":"",
"axisPlacement":"auto",
"barAlignment":0,
"drawStyle":"line",
"fillOpacity":0,
"gradientMode":"none",
"hideFrom":{
"legend":false,
"tooltip":false,
"viz":false
},
"insertNulls":false,
"lineInterpolation":"linear",
"lineWidth":1,
"pointSize":4,
"scaleDistribution":{
"type":"linear"
},
"showPoints":"auto",
"spanNulls":false,
"stacking":{
"group":"A",
"mode":"none"
},
"thresholdsStyle":{
"mode":"off"
}
},
"mappings":[],
"thresholds":{
"mode":"absolute",
"steps":[
{
"color":"green",
"value":null
}
]
},
"unit":"percentunit",
"unitScale":true
},
"overrides":[]
},
"gridPos":{
"h":8,
"w":12,
"x":0,
"y":164
},
"id":102,
"links":[],
"options":{
"legend":{
"calcs":[],
"displayMode":"list",
"placement":"bottom",
"showLegend":true
},
"tooltip":{
"mode":"single",
"sort":"none"
}
},
"pluginVersion":"8.1.2",
"targets":[
{
"datasource":{
"type":"prometheus",
"uid":"${DS_PROMETHEUS}"
},
"editorMode":"code",
"expr":"(sum by (cluster, namespace, job, instance) (rate(SeaweedFS_vector_cache_requests_total{namespace=~\"$NAMESPACE\",result=\"hit\"}[$__rate_interval])) or on (cluster, namespace, job, instance) (0 * sum by (cluster, namespace, job, instance) (rate(SeaweedFS_vector_cache_requests_total{namespace=~\"$NAMESPACE\"}[$__rate_interval])))) / sum by (cluster, namespace, job, instance) (rate(SeaweedFS_vector_cache_requests_total{namespace=~\"$NAMESPACE\"}[$__rate_interval]))",
Loaded 100 of 1831 files, more files were not shown because too many files have changed in this diff.
Show more
Reference in new issue
Block a user
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.