With jwt.signing.read set, volume servers refuse unsigned reads, so
cp.from of a chunked file failed with "source returned status 401".
Mint a read token for the chunk on each source attempt, the same way the
filer's own reads do.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A follower took the scheme from [https.client], so with HTTPS clients
toward volume servers and plain-HTTP masters every forwarded request
(/dir/assign, /dir/status, the UI) failed with 502 "first record does not
look like a TLS handshake". Masters share one [https.master] setting, so
use HTTPS exactly when this master serves it.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
security.toml is mounted with subPath and read once at startup, so on an
upgrade that turns volume HTTPS on, the volume servers restarted with TLS
while filer, S3 and the other clients kept speaking plain HTTP. A
checksum/volume-https pod annotation, rendered only while the option is
on, rolls them all.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The copy built a bare client and plain http:// chunk URLs, so with
[https.client] on it could not read from or write to HTTPS volume servers
("source returned status 400"). Use the global client's transport and
redirect policy, and normalize the source and destination URLs to its
scheme.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Its volume server registers by pod IP, so HTTPS clients could not verify
the volume certificate.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Volume servers register as <pod>.<volume service>.<namespace>, and HTTPS
clients verify that name, which none of the certificate's wildcards cover.
With volume HTTPS on, the certificate also lists every volume group's
service.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
global.seaweedfs.volumeHttps.enabled sets [https.volume] to the volume
certificate and turns on [https.client] with the client certificate and the
chart CA, so the volume data port is no longer plaintext under
enableSecurity. The option fails without enableSecurity.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A scheduled run whose jobs no executor can take gives them up at once,
leaving them to the next detection. For a job type with a long detection
interval that is a whole interval later: a daily job whose only worker
was briefly full lost a day.
Bring the job type's next detection forward to within 10 minutes when
its jobs are given up for capacity, never pushing back an earlier one.
Claude-Session: https://claude.ai/code/session_011UpvafcfUuC2mLxNAXLQJi
* filer: renew AppendToEntry's cluster lock while the append runs
AppendToEntry took a 5 s short-lived lock and never renewed it, so an
append slower than that (slow store, slow volume, GC pause) lost the
lock while still running, and another filer could take it and append
to the same entry concurrently, dropping one append's chunks. Use the
auto-renewing lock the HTTP append path already uses.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* filer: refuse an AppendToEntry commit once its cluster lock was lost
A renewal can still fail, or the lock can be lost and granted again
under a new generation, while the append runs; another filer may have
appended in between. Check the lock and its generation right before
the entry is written, fail the append so the caller retries, and drop
the manifest chunks this attempt created.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* cluster: IsLocked reports a lock whose lease ran out as not held
isLocked only flipped when a renewal came back rejected, a second after
the reply, so a holder whose renewal stalled past the lease (a long GC
pause) still read as locked while the server could hand the key to
someone else. Track the lease from each grant's send time and require
it in IsLocked, which the filer append paths check before committing.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* filer: hold the slow-append test past the renewing lock's first lease
The renewing lock grants a 7 s lease, so a 6 s hold passed even with
renewal off; wait past LiveLockTTL so only a renewal keeps the lock.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* filer: say the AppendToEntry lock check is best effort
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* s3vectors: render filter string control characters as serde_json does
json_extract serializes the stored value with serde_json: \b, \f, \n,
\r, \t, \u00XX for the rest of U+0000-U+001F, and U+007F-U+009F raw.
The scalar comparison text used Rust debug escapes (\0, \u{1}, \u{7f}),
so $eq/$in on such strings missed scalar values and $ne/$nin kept them.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* test(s3vectors): filter on every control character, quote and backslash
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* vector: derive the indexed metadata key columns
An indexed metadata key is a hidden `_mk_<n>` column of List<Utf8>: the
strings an S3 `$eq` on a string matches, that is the value when it is a
string or the string elements of an array, and nothing for any other type
or a missing key. The field metadata names the key and its state
(`building` or `ready`).
The module derives the column from metadata given as JSON (the write
path) or as stored (jsonb or JSON text, for a backfill), and reads the
indexed keys from a schema. It lives in the worker crate so the engine and
the worker share it.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* vector: fill the indexed metadata key columns on PutVectors
PutVectors builds its batch against the table's current schema. Once a
table has indexed metadata key columns, the three-column batch no longer
matches that schema and the write fails. Fill each `_mk_*` column the
schema has from the row's metadata, whatever the key's state; a column
the engine does not know still fails the request.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* vector: answer string $eq and $in on a ready indexed key from its column
The gateway renders a string `$eq` or `$in` on a key as an OR of
`json_extract(metadata, P) = '"v"'` (or IN) and
`json_array_contains(metadata, P, '"v"')` over the same values. On the
parsed filter, a top-level conjunct of that shape on a key whose column is
`ready` becomes `array_has` / `array_has_any` on the column, which its
LABEL_LIST index answers. The two halves must name the same key and the
same strings, the scalar half in the text json_extract returns for them;
anything else, including `$ne`, `$nin`, `$exists`, numbers and booleans,
stays on the JSON path, and the other conjuncts still apply.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* vector: score few indexed matches exactly, per scope
With a ready indexed key in the filter, `ann` first counts the rows of its
scope that the key's conjuncts match, through the scalar index. When they
are at most the limit it scores them exactly over that scope, the way
the short-result fallback does; above it, the vector index searches with
the indexed prefilter, and the fallback still applies. The scope is the
one `exact_in_scope` scores, so per-scope exact answers merge into the
global exact top-k.
The limit is `Query::exact_match_limit` (None takes 256; 0 never counts),
set by the engine's --exact-match-limit (WEED_VECTOR_EXACT_MATCH_LIMIT).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* vector: count filtered queries by metadata key
For each index, the engine counts its QueryVectors calls by the metadata
keys of the parsed filter, with their total latency and the operand types
each key was compared with, over the last hour in 5-minute buckets. At most
512 indexes and 32 keys per index are kept (an idle key makes room;
otherwise the index is marked truncated), and the stats are not exported
as metrics labels.
The new FilterStats RPC returns an index's stats; the gateway's discovery
client forwards it like the other calls.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* vector: report the metadata columns capability in gRPC health
An engine whose writes fill the indexed metadata key columns reports the
gRPC health service `weed.vector.capability.metadata_columns` as serving,
so an operator action can refuse to index a key while an older engine
could still write the table.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* vector: expire filter stats' operand kinds and truncation with the window
The operand kinds a key was compared with, and the mark that a key went
uncounted, outlived the queries that set them. Keep the kinds per
5-minute bucket and the truncation as the bucket it last happened in, so
both leave the hour window with the counts.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* vector: fill indexed key columns a write left empty racing their add
A PutVectors that opened the table before an indexed key's column was
added, and commits after the add, is rebased by Lance onto the wider
schema: it commits, and the rows it wrote hold NULL in the new column, so
the column's index misses them (reproduced in the test: 100 of 150 rows).
After its commit, PutVectors now looks for indexed-key columns the batch
did not carry, reads those of its keys whose column is still NULL, derives
the values from their stored metadata and patches just those columns with
a partial-schema merge that updates only rows still NULL.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* vector: keep filter stats' truncation mark at the later bucket
record() reads the clock before it takes the lock, so a call that waited
on the lock across a bucket boundary could move the truncation mark (and
the index's last-used bucket) back to its earlier bucket, and the mark
then left the window one bucket early.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* vector: derive a raced write's indexed key columns from current metadata
A PutVectors that opened the table before an indexed key's column was
added is retried by Lance against the wider schema: rows it inserts hold
NULL there, and rows it updates keep the value derived from their old
metadata, which the NULL-only heal never looked at. The heal now derives
the columns for all of the write's keys.
Its merge also ran with Lance's retry, which joins the values derived
before the conflict onto the rows a later write of the same keys left.
It now fails on a conflict and is redone from the latest version.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* vector: check the raced write's keys keep the later write's rows
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* volume: apply a tailed size-0 record as a delete when the key exists
A delete and an empty write both reach the tail receiver as a size-0
.dat record. Writing every one as a live needle turned each delete
caught by volume.move's tail into a 200-empty object. Treat the record
as a delete when the key already has a record on the receiver, and as
an empty write otherwise.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* rust volume: apply a tailed size-0 record as a delete when the key exists
Mirrors the Go tail receiver: a size-0 record for a key that already
has a record here is a delete, otherwise an empty write.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* shell: volume.merge applies a size-0 record as a delete when the key exists
The tail stream carries deletes as size-0 records, and volume.merge
wrote each one to the merged copy as a live empty needle, so every
delete in the volume came back as a 200-empty object on all replicas.
A size-0 record for a key already written in the merged stream is now
a delete on the target; for a new key it stays an empty write.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* shell: volume.merge reads a size-0 record as a delete only after its own replica's record
Replicas append the same write at their own times, so a fresh empty write
can reach the merge twice when the copies are further apart than the
dedup window. Tracking keys across all replicas made the second copy a
delete. Track, per source replica, the keys it has returned (duplicates
the merge skips included): a size-0 record is a delete only when its own
replica wrote the key before, which is the only way a replica appends a
delete. A delete for a key the merged copy already deleted sends nothing.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* shell: volume.merge verification counts the deletes the merge applied
Applying deletes lets a correct merge hold fewer live needles than a
replica that missed a delete, and the live-count check then refused it
and kept the divergent replicas. Every key live on a replica is either
live or deleted in the merged copy, so compare that replica's live count
with the merged live count plus the keys the merge deleted. An empty
merged copy is still refused.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* volume: tail receiver looks a size-0 record's key up in the needle map
The receiver read the needle to decide whether a size-0 record deletes
an existing key. A read reports a needle past its TTL as not found, so
a delete for it was written as a live empty needle, which reads ignore
TTL for. Look the key up in the needle map instead (Volume.LookupNeedle).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* rust volume: tail receiver looks a size-0 record's key up in the needle map
Mirror Go: write_tailed_needle read the needle to decide whether a size-0
record deletes an existing key, and a read reports a needle past its TTL
as not found, so the delete became a live empty needle. Look the key up
in the needle map instead (Volume::lookup_needle).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* shell: volume.merge keeps a merged delete when a late empty copy follows
A replica that missed a delete can hold its copy of the empty write
stamped after the other replica's delete. As that replica's first record
for the key it was written, reviving the deleted key. A size-0 record now
never revives a key the merged copy already deleted.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* shell: volume.merge refuses more replicas than its key history has bits
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* wdclient: keep a volume's live replicas readable while another volume cannot be found
A filer's "volume not found" answer counted as a filer failure, so three
of them opened the circuit breaker on a healthy filer. With its only filer
in the cool-down, every lookup then failed without asking, as "all 1
filer(s) failed with non-retryable error: %!w(<nil>)". And a refresh whose
lookup failed left the cache without the volume's live replicas, since
InvalidateCache had already dropped them with the dead one.
The breaker now counts only failures to reach the filer, and asks the
filers anyway when all of them are in their cool-down. InvalidateCache keeps
what it removed; a lookup that errors puts it back unless the volume was
learned again, the epoch moved, or a reset would have expired it, and
leaves out locations fenced by a pushed removal.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* wdclient: set invalidated locations aside in the map instead of copying them out
The first commit kept what InvalidateCache removed in a side map and put it
back when the lookup failed. The copy left the map's lock, so a removal the
master pushed between the take and the put-back was undone: the removed
server came back into the cache.
The entry now stays in the map with a state. entryStale hides it from readers
and releases its server references, as a delete did. A pushed removal edits
it like any other entry. When the lookup after the invalidation returns an
error, the entry becomes entryRevived and is served again.
A stale or revived entry holds only what was known before the read failed,
so the next thing learned replaces it: a lookup answer drops it before the
answer is cached, including an answer with no locations and an answer to a
lookup that was in flight when another one failed, and a pushed add replaces
it instead of merging.
LookupVolumeIdsWithFallback returns no error when every volume it was asked
for has locations after the revive, so the filer's LookupVolume handler sends
its response instead of an error status.
The not-found breaker test now checks the failure count itself. Without
that, a regression of the classification passed, because a lone filer in its
cool-down is asked anyway.
On the first commit, TestPushedRemovalRacingFailedRefreshes failed in 8 of 8
runs under -race: four with "got [live:8080 old:8080], want [live:8080]",
four with "got [], want [live:8080]".
Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
* wdclient: retry a filer lookup pass that asked no filer
The pass decides once whether to skip unhealthy filers. A filer whose
breaker trips after that decision, as when a concurrent lookup records
its failure, is still skipped. With every filer skipped the pass ended
with no error and failed the lookup as non-retryable with
"%!w(<nil>)". Such a pass now counts as Unavailable, so the lookup
retries and the next pass asks the filers anyway.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* wdclient: believe a provider answer that reports errors for some volumes
The master provider answers a batch with a map of the volumes it knows
and an error naming the ones it does not. The failed-lookup branch took
that as no answer and revived what InvalidateCache had set aside, so a
volume the master reported as absent came back with its old locations,
and the batch could return them with a nil error. Only a nil map now
means no answer. Otherwise the answer replaces what was set aside, and
the provider's error is still returned.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* wdclient: keep set-aside locations learned after a lookup was asked
A successful lookup dropped every stale or revived entry of the volumes
it asked for, however old the answer was. A lookup asked before the
master confirmed new locations could, on finishing, delete those
locations after a read had set them aside and a failed refresh had
revived them, and its own answer was not cached because the epoch had
moved. Entries now record when their locations were learned, and an
answer drops only what was learned before it was asked.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* wdclient: keep an older answer from replacing locations set aside after it was asked
fdcde25 keeps dropSetAside from deleting set-aside locations that were
learned after a lookup was asked. The answer's insertion still replaced
them: fillLocations adds each location with vidMap.addLocation, and
addLocationToMap replaces a stale or revived entry whatever its learnedAt.
The epoch does not stop this when the newer locations came from another
lookup's answer or, on the master client, from a pushed add, since neither
moves it. The master client's provider also caches its answer itself,
before LookupVolumeIdsWithFallback reaches dropSetAside.
vidMap.addAnswer adds an answer's locations under one lock, and adds none
of them while vid has a stale or revived entry learned after the lookup was
asked. fillLocations uses it, and the master client's provider passes the
learned mark it takes next to its epoch. An answer with nothing newer in the
way still replaces what was set aside, with every location it reports.
On fdcde25, TestOlderAnswerDoesNotReplaceNewerRevivedLocations and the
"newer location revived during the lookup" case of
TestMasterAnswerAgainstSetAsideLocations fail with
"got [old:8080], want [new:8080]".
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* wdclient: move addAnswer next to dropSetAside
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
* admin/plugin: fail a restored job its worker no longer lists
An admin restart reloads the jobs that were running from the persisted
state, but the workers drop those jobs when the old admin stream closes.
No pending execution covers a restored job, so only the 24h stale sweep
ended it, and until then it blocked dedupe and scheduling of its job type.
Track the active jobs loaded at startup by worker and fail one after two
itemized heartbeats from its worker leave it out, as is already done for
a dispatch whose worker restarted. A listed job is left to adoption.
Claude-Session: https://claude.ai/code/session_011UpvafcfUuC2mLxNAXLQJi
* admin/plugin: keep tracking a listed restored job; skip queued ones
- A restored job its worker lists only resets its miss count, so one it
drops later is still failed.
- Track only jobs that were sent to a worker (request id set, not pending).
- Match restored ids against the trimmed heartbeat ids.
- Fail the restored attempt under the jobs lock that applies the expiry
(handleJobCompletedIf / expireActiveJobIf), so a newer attempt of the
same job is neither failed nor canceled.
Claude-Session: https://claude.ai/code/session_011UpvafcfUuC2mLxNAXLQJi
* admin/plugin: keep a newer attempt's detail from a restored-job expiry
- The conditional completion writes the job detail through
persistJobDetailSnapshotIf and skips it when the stored detail already
names another request; the check runs under jobDetailsMu with the write.
- mergeTerminalDetailIntoTracked ignores a terminal detail of another
request, so a restart does not end the loaded attempt with it.
- Tests wait for the restored-job expiries instead of sleeping.
Claude-Session: https://claude.ai/code/session_011UpvafcfUuC2mLxNAXLQJi
* admin/plugin: restore a finished retry from its detail
The job detail is saved as each attempt changes and the job list only
periodically, so a terminal detail naming another request is the newer
attempt. Apply it with its request id instead of ignoring it; the
expiry's own detail write already refuses to overwrite another request.
Claude-Session: https://claude.ai/code/session_011UpvafcfUuC2mLxNAXLQJi
* admin/plugin: restore whichever attempt is newer
The job list and a job's detail can name different attempts either way
round: the list is saved periodically, and a detail write can fail. When
their request ids differ, apply the terminal detail only if it is the
newer attempt (higher attempt number, else later update).
Claude-Session: https://claude.ai/code/session_011UpvafcfUuC2mLxNAXLQJi
* admin/plugin: fill a restored job's empty request id from its detail
A job the list saved while still queued has no request id; take the
finished detail's, as the merge did before the newer-attempt check.
Claude-Session: https://claude.ai/code/session_011UpvafcfUuC2mLxNAXLQJi
* admin/plugin: take the worker and attempt of a newer restored detail
When the restore takes a finished retry's detail over the job list's
attempt, the record kept the older attempt's worker and attempt number
next to the retry's request id and state.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* admin/plugin: fold the conditional completion into the existing functions
handleJobCompleted, expireActiveJob, trackExecutionCompletion and
persistJobDetailSnapshot take the accept check themselves instead of
gaining *If twins.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* admin/plugin: ignore a late earlier-attempt completion; replace an older detail
- A completion whose request id differs from the tracked attempt's is a
late report of an earlier attempt and no longer ends a retry.
- A conditional completion skips the detail write only when the stored
detail is a newer attempt (detailIsNewerAttempt), so an older finished
attempt's detail left behind by a failed write is replaced.
Claude-Session: https://claude.ai/code/session_011UpvafcfUuC2mLxNAXLQJi
* admin/plugin: take only the dispatched attempt's reports
A late progress report of a timed-out attempt replaced the retry's request
id, after which the retry's own completion was rejected as foreign. One
rule now gates both: a record the admin dispatched (attempt > 0) takes
progress and completion only for its own request (reportFromOtherAttempt).
When an expiry replaces an older attempt's detail it also takes the
closing attempt's worker, attempt number and times.
Claude-Session: https://claude.ai/code/session_011UpvafcfUuC2mLxNAXLQJi
* admin/plugin: restore a newer running attempt's identity from its detail
The job list can name attempt A while the detail shows retry B running
when the admin stopped. The restore now takes B's request id, worker and
attempt number from such a newer running detail, so B's own reports are
accepted. Test helper keeps a given attempt number.
Claude-Session: https://claude.ai/code/session_011UpvafcfUuC2mLxNAXLQJi
* admin/plugin: keep attemptEnded's doc comment on attemptEnded
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* admin/plugin: dispatch an attempt-less job as attempt 1
The late-report guard only covers records with a positive attempt, and
ExecuteJob passed a zero attempt through, so a direct caller's dispatch
could be closed by another request's completion. The HTTP API already
raised zero to one; the dispatch path now does the same for every caller.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* filer: keep a hard-linked file's chunks when one of its names is replaced
Replacing one name of a hard-linked file deleted the file's chunks while
its other names still pointed at them. Reads through those names then
return EIO once the volumes are vacuumed; until then they only work
through readDeleted.
Two everyday ways to get there through a mount:
- conda links an environment's files to its package cache with hard
links. Python saves a __pycache__/*.pyc by writing a temporary file and
renaming it over the old name, so the cache's copy loses its data.
- A training script saves model.pt, hard-links it into an epoch directory
and saves the next epoch by renaming a new file over model.pt. Each
archived epoch loses its data.
Three write paths deleted the chunks of the entry they replaced without
asking whether that entry was one of several hard links:
- a rename over a file: moveSelfEntry drops the target's link, then
queues all of the target's chunks for deletion;
- CreateEntry over an existing path (an S3 PUT, for one): the update
drops the old entry's link, then deleteChunksIfNotNew deletes its chunks,
and the CreateEntry handler deletes chunks its own list covers;
- UpdateEntry with an entry that no longer carries the old one's link.
A plain delete already keeps the chunks while HardLinkCounter > 1, and a
folder delete asks hardLinkIsOrphaned. The three paths now go through
Filer.KeepHardLinkedChunks, which asks after the write, once the replaced
name's link is gone: the chunks stay unless the file's shared record went
with it, that is, unless the replaced name was the last link. Writing the
file through one of its names (an entry with the same hard link id) still
deletes the chunks the write replaced, since every name reads the new
content.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* filer: keep hard-linked chunks from the shared record, not the replaced row
A lookup that cannot read the hard link record returns the name's own row,
which can lag the file. Read the record when filtering the garbage, and keep
everything when it cannot be read.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* filer: test the manifest reads behind KeepHardLinkedChunks
The manifest resolution moves into withoutChunksOf, which takes the lookup
function, so tests can serve manifests. Chunks behind nested manifests and
the manifests themselves stay, unrelated garbage is still deleted, and an
unreadable manifest returns an error, on which KeepHardLinkedChunks deletes
nothing. Behavior is unchanged.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
* volume: count needles vacuum drops as unreadable and log each at error level
Vacuum drops a live needle whose record fails to read back intact (CRC
mismatch, malformed fields, short or dangling record). The key leaves the
compacted index with no tombstone, so repair tools later treat it as a
vacuumed delete. Log every dropped key at error level and count the drops
in SeaweedFS_volumeServer_vacuuming_dropped_needles per collection, once
the compacted index is written.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* grafana: add a panel for needles dropped by vacuum
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* seaweed-volume: count needles vacuum drops as unreadable and log each at error level
Mirror the Go volume server: log every dropped key at error level and add
SeaweedFS_volumeServer_vacuuming_dropped_needles per collection once the
compacted index is written.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* volume: count dropped needles only after the compacted dat is flushed
The final .cpd fsync runs in the deferred close and loadTmpNeedleMap can
still fail after the index is saved; a copy failing there is retried and
would count the same needles again.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* filer: lock the path for plain HTTP overwrites
CreateEntry looks up the entry it replaces and queues that entry's chunks
for deletion. Two plain PUT/POST writes of one path could both read the
same old entry, so the chunks of whichever write lost were never queued.
Take the exclusive path lock around CreateEntry, as the gRPC CreateEntry
already does.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* filer: test concurrent HTTP overwrites queue every replaced chunk
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* filer: resolve deleted manifests through the master on a volume cache miss
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* filer: keep a manifest needle until its sub-chunks are queued for deletion
When a manifest could not be read during chunk deletion, the manifest
needle was deleted anyway and the data chunks it listed were orphaned.
Retry the resolve with the deletion retry backoff and delete the
manifest alone only after MaxRetryAttempts.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* filer: keep a durable delete that joins a pending manifest retry
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* filer: bound each manifest retry read with a deadline
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* filer: requeue resolved manifest ids on an fsync upgrade instead of rereading
A durable delete that joins a manifest retry in flight used to resolve
the manifest a second time, but the first pass had already queued the
manifest needle, so the reread could fail and the sub-chunks never got
the durable mark. Resolve nested manifests up front and queue the same
ids again with fsync.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* filer: bound each manifest retry pass and stop it on shutdown
A pass takes ready manifests one at a time and stops after
manifestRetryTimeout, leaving the rest queued for the next tick, so a run
of stalled reads cannot keep the shared retry loop from ordinary deletion
retries or from noticing shutdown.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* http: report the status of a failed delete that has no body
A replica that answers a DELETE with a bare 500 left the forwarding
server reporting "[<replica>]: " with an empty reason.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* volume: forward a delete retry that finds nothing here to the replicas
A delete is applied locally before it is sent to the other replicas, so
when the fan-out fails the client is told to retry while this replica
already dropped the needle. The retry then read nothing here and answered
404 without asking the replicas, so the replica that missed the delete
kept the needle for good.
On a replicated volume, a plain delete that finds the needle missing or
deleted now cancels a write still queued for it, as a replica delete
does, and sends the delete on to the other replicas. A failed fan-out is
reported as before, so the client retries until every replica agrees.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* volume(rs): take a replica's 404 to a forwarded delete as done
A replica that no longer holds the needle answers a forwarded delete with
404. The Go server folds that into success, but the Rust server reported
it as a replication failure, so a delete that only had to catch up one
replica failed on the replica that was already in step.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* volume(rs): forward a delete retry that finds nothing here to the replicas
Mirrors the Go server: on a replicated volume, a plain delete that finds
the needle missing or deleted cancels a write still queued for it and
sends the delete on to the other replicas, so a retry after a failed
fan-out reaches the replica that kept the needle.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* volume: answer a delete whose needle read fails with 500, not 404
A replica's 404 to a forwarded delete is taken as the needle being gone
there. DeleteHandler answered 404 for any failed pre-read, so a replica
whose disk could not read the needle let the primary acknowledge a delete
that replica never applied. Only a missing or deleted needle (or a volume
this server does not hold) answers 404 now; other read errors are a 500,
so the client retries.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* volume(rs): answer a delete whose needle read fails with 500, not 404
Mirrors the Go server: a replica's 404 to a forwarded delete is taken as
the needle being gone there, so only a missing or deleted needle answers
404; other read errors are a 500 and the client retries.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* volume(rs): corrupt the needle with portable file writes in the delete read test
FileExt::write_all_at is Unix-only, so the test did not build on Windows.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* volume: serve decoded bytes for a range over a compressed needle
A Range with Accept-Encoding: gzip got Content-Encoding: gzip and a slice
of the compressed bytes, which a decoding client cannot read. The range
is over the stored file's decoded bytes, so pass gzip through only for a
whole read.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* seaweed-volume: serve decoded bytes for a range over a compressed needle
Mirrors the Go volume server: a Range with Accept-Encoding: gzip got
Content-Encoding: gzip and a slice of the compressed bytes. Pass gzip
through only for a whole read.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* seaweed-volume: HEAD of a compressed needle reports the served length
HEAD answered a compressed needle from its meta: the stored gzip length
with no Content-Encoding, whatever the request accepted. Go reads the
payload and reports what the GET would send: the decoded length, or the
gzip length with Content-Encoding: gzip for a whole read that accepts
gzip. Read a compressed needle in full on HEAD too.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* test/volume_server: range and HEAD over a compressed needle
Both volume servers serve a gzip-accepted range decoded, and HEAD reports
the length the matching GET would send.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* seaweed-volume: decode every gzip member of a compressed needle
Go's gzip reader reads concatenated members; GzDecoder stopped after the
first, so a range over such a needle saw a shorter body and length.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* vector: answer a short filtered ANN query by an exact scan
A prefiltered query through the IVF index sees only the partitions it
probes. When those hold fewer than k eligible rows the answer is short
(400 matches, topK 500, 63 partitions: about 208 returned), and when at
most k rows match Lance appends the ones its probes missed with an
infinite distance. Either case is now answered again by an exact scan of
the same scope: the listed fragments plus the live coverage of the listed
index segments, or the whole table.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* vector: count an explicit probe count as indexed only with an index
With nprobes set on a table that has no vector index, the scan was marked
indexed, so a filtered query matching fewer than k rows ran the same exact
scan a second time.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* filer: retry TiKV store writes on a write conflict
TiKV transactions are optimistic. A commit fails with a write conflict
when another transaction committed one of its keys after it began, even
for a blind Set or Delete. Outside a filer transaction the TiKV store
runs each save and delete in a transaction of its own and returned that
error to the caller, so of two requests writing the same entry at the
same moment one failed, and a mount reported EIO.
A write the store commits itself (InsertEntry, UpdateEntry, DeleteEntry,
one batch of DeleteFolderChildren, KvPut, KvDelete) now runs again in a
new transaction after a write conflict, up to 5 times, waiting 1-2ms,
2-4ms, ... 16-32ms in between. The failed transaction stored nothing,
and the retry lands after the other writer, as if the two had not
overlapped. The retries stop when the context is done.
A write inside a transaction from BeginTransaction is not retried: the
conflict is reported when the caller commits, as before.
deleteBatch now goes through RunInTxn like the other writes.
Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
* filer: stop TiKV write retries once the context is done
When the context was cancelled while the retry wait was running, the
timer case could still win the select and start one more transaction.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* filer: keep a TiKV entry that is no longer expired when listing deletes it
The listing decides an entry has expired from its own read, and the
delete ran later as a blind delete. Once the delete retries after a
write conflict, a live replacement committed in between was removed.
Read the entry again in the deleting transaction and delete it only if
it is still expired.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: MapleWhiteZZZ <18653045+MapleWhiteZZZ@users.noreply.github.com>
Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
* storage: reject a streamed read whose record id or size differs
The header probe before a streamed read returned not-found only when both
the record's id and its size differed from the index entry, so a record of
another needle with the same size passed and its data was streamed under
the requested key until the final checksum check.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* storage: refuse a needle blob whose header names another needle
WriteNeedleBlob compared only the blob header's size with the size it was
asked to index. A blob read from an offset that has since moved to another
record of the same size was appended and indexed under the wrong key.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* volume(rs): refuse a needle blob whose header names another needle
Mirror of the Go check: write_needle_blob_and_index compared only the blob
header's size, so a blob of another needle with the same size was appended
and indexed under the requested key.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* volume(rs): report not-found when a streamed read's record is another needle
The meta read ahead of a streamed reply took the id from the record header
without comparing it with the requested key. Return not-found when the
index entry resolves to another needle, as the Go header probe does.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* storage: check the requested id after a meta-only read
ReadNeedleMeta overwrites n.Id with the record's header id, so a meta-only
read through an index entry that resolves to another needle returned that
needle's metadata, and the streamed-read probe that follows compared the
record with itself. Return not-found when the header names another needle,
as the Rust meta read does.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* test: copy a needle blob to another volume, not under another key
WriteNeedleBlob now refuses a blob whose header names another needle,
so the round trip writes the blob under its own id into a second volume
and checks that storing it under another key is refused.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* storage: compare the streamed header with a deleted entry's record size
A deleted index entry stores the negated size, so a read-deleted stream
compared the positive header size with a negative one and missed.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* storage: copy an empty needle written during compaction in makeupDiff
A size-0 index row is a live empty needle, not a delete. makeupDiff
grouped it with tombstones and appended a made-up delete marker, so
the compacted .dat lost the record's cookie and held a delete.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* volume(rs): copy an empty needle written during compaction in makeup_diff
A size-0 index row is a live empty needle. makeup_diff wrote a made-up
delete marker and a tombstone row for it, so the needle read as not
found after the commit. Copy the record like any other live row.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* storage: fail makeupDiff when the replayed needle write fails
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* volume(rs): check the compacted empty needle's id and cookie in the test
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* admin: show a pruned job's detail from its run record
Tracking keeps the newest 1000 jobs across all job types and deletes
the details of older finished ones, while each job type's run history
keeps its last 10 successful and 10 failed runs. A job listed in the
run history could therefore open as "job detail not found". Fall back
to the run record when the job is no longer tracked.
Claude-Session: https://claude.ai/code/session_011UpvafcfUuC2mLxNAXLQJi
* admin: show 100% progress for a pruned successful run
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* admin: use the newest run record for a job across both outcomes
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* admin: keep the run record a pruned job's detail was built from
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* admin: keep looking for a run record past an unreadable history
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* admin: pick a pruned job's newest run record across job types
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* storage: red test for CommitCompact after CompactByVolumeData with a concurrent write or delete
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* storage: build the temporary needle map in CompactByVolumeData
Fixes#12783
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* storage: require a successful commit in the CompactByVolumeData commit test
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
BinarySearchByAppendAtNs, its left/right probes and ResumeByLastEntry read
every probed record through readNeedleAt, which parsed the whole body and
verified the data checksum, although they use only the id, cookie, size and
append time. One record with damaged data on the search path, even a needle
deleted long ago, failed the search and with it the tail sender, a
timestamp-based incremental copy and the volume.move catch-up.
readNeedleAt now reads the header and the trailer and skips the data. The
data checksum is still verified where the data is read, including the tail
receiver for every record it gets. The Rust volume server already parses
only the metadata on this path and does not verify the data checksum.
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* storage: red test for the append-time search racing a vacuum commit
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* storage: hold the data-file read lock across the append-time search
Fixes#12785
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* storage: Version() writes the superblock version only when it changes
Version() stored the .vif version into SuperBlock.Version on every call, so
each call was a write that raced with readers of the superblock that do not
take superBlockAccessLock, such as the append-time search and the
compaction setup.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* storage: harden the append-time search race test; add a lock-hold budget test
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* storage: Linux-only lock budget; stop before reading search failures
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
* security: add a constant-time HTTP Basic Auth check
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* filer: require HTTP Basic Auth on the HTTP ports when configured
With filer.basic_auth.password set in security.toml (or
WEED_FILER_BASIC_AUTH_PASSWORD), a request on the filer HTTP ports, TUS
included, must carry those credentials or a valid filer JWT, and an
access level with no JWT signing key is no longer open. Valid Basic
credentials grant read and write. 401 responses then carry a Basic
challenge so browsers prompt. The credentials reload on SIGHUP with the
signing keys.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* scaffold: document filer.basic_auth in security.toml
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* filer: enable reloaded Basic Auth before swapping the JWT keys
Reload cleared the old signing keys before storing the new Basic
credentials, so a switch from JWT to Basic Auth briefly left both off
and admitted anonymous requests. Store new credentials first and clear
old ones last.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* filer: refuse Basic credentials on cross-origin browser requests
Browsers resend cached Basic credentials on requests other websites
trigger, such as a form POST with mv.from, and the default
cors.allowed_origins "*" lets them through. Accept Basic credentials
only on requests without an Origin, from the filer's own host, or from
an origin explicitly listed in cors.allowed_origins. A JWT is still
accepted from any origin.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* filer: recheck Basic Auth after reading an empty signing key
A request that loaded Basic Auth as off, then read the signing key after
a JWT-to-Basic Reload had cleared it, was let through anonymously. Open
access now also needs Basic Auth to be off after the key read; with
Reload enabling the new credential before disabling the old one, the
two loads cannot both miss a credential that was on throughout.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* security: compare Basic Auth credentials as MACs under a per-process key
A plain SHA-256 of the password is what password-hash scanners flag;
keying the digests with a random per-process HMAC key keeps the
length-independent constant-time compare without holding one.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* util: read a configuration file without merging into the shared viper
LoadConfiguration merges each read into the shared viper, so a key
removed from the file keeps its old value after a reload.
ReadConfiguration reads the file into a fresh viper with the same search
path and WEED_ environment overrides.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* filer: turn Basic Auth off when security.toml no longer sets it
The shared viper merges each SIGHUP reload into what it held, so
removing [filer.basic_auth] kept the old password valid. Read the
credentials from security.toml alone at startup and on reload; an
unreadable file keeps the current setting.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* filer: keep Basic Auth when security.toml cannot be read
A malformed file at startup dropped a WEED_FILER_BASIC_AUTH_PASSWORD
from the environment, and viper reports an unsearchable config
directory as a missing file, which turned Basic Auth off on reload.
ReadConfiguration now always returns the environment overrides and
reports whether the file was read; a reload that could not read it
keeps the current credentials.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* filer: keep reloading Basic Auth from the security.toml first read
A search directory that turned unsearchable made viper fall through to a
lower-priority security.toml, and a reload from a file without
[filer.basic_auth] turned Basic Auth off. ReadConfiguration now takes the
file to read, and the filer reads the file it first found on every reload.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* wdclient: sort newly discovered filers and log every current-filer switch
Discovery appended new filers in map iteration order, so after the current
filer left, the fallback at index 0 was a random filer. Append them sorted.
Route every change of the current filer through one helper that logs it at
V(0) with its reason (failover, return_home, discovery), so an operator can
see which filer a gateway uses without new metrics.
* wdclient: prefer a home filer and return to it after failover with backoff
A FilerClient with FilerClientOption.HomeFiler set goes back to that filer
once its circuit breaker allows a re-check, instead of staying on whichever
peer it failed over to. Leaving home again soon after a return doubles the
wait before the next return, up to 5 minutes; a return that holds for 10
minutes resets it. Volume lookups and discovery follow the same preference.
* s3: embedded gateway uses its local filer as home filer
weed filer -s3, weed server -s3 and weed mini now build the gateway's filer
address in the form the filer registers in the master (ip:port.grpcPort),
with the gRPC port default applied, and pass it to the gateway as its home
filer, so the gateway returns to its local filer after any failover.
Standalone weed s3 has no home filer and behaves as before.
Refs #12768, follow-up to #12771
* wdclient: switch the current filer only from the filer a request started on
A volume lookup that started on a peer and finished after the client had
returned home moved it back to the peer and counted that as leaving home,
doubling the return backoff although home never failed. Concurrent
failovers could also each count as a separate departure.
Failovers now name the filer they started on and take effect only while it
is still current, via compare-and-swap, so only the winning switch logs and
feeds the backoff. S3's failover walk uses the same rule.
* wdclient: switch the current filer and record the return backoff in one step
The index moved by compare-and-swap before the switch was recorded, and
both ran under the shared read lock. A request reading the current filer
in that gap saw the peer with the old eligibility time and, with home's
breaker still closed after one failure, switched straight back home.
The eligibility check, the move and the bookkeeping now all run under
homeMu.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* wdclient: discovery falls back past a home that is still backing off
When discovery removed the current filer and home was not yet eligible,
the client took index 0, which is home whenever it is listed first. With
home's breaker still closed after a single failure, requests went back to
the filer they had just left. Take the next filer instead.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* wdclient: start return timing on every switch back to the home filer
Only a return_home switch stamped lastReturnHome. Discovery or a peer's
failover could also put the client back on home, and a failure soon
after such a return kept the backoff at its base value instead of
doubling it.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Konstantin Lebedev <whitefox@mayflower.work>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* filer: recover leveldb without rebuilding broken tables
leveldb.Recover rebuilds a table with damaged blocks using the user
comparer and filter on internal keys. The shortened index keys it writes
are not valid internal keys, so a Get on the rebuilt table panics, and
the bloom filter it writes misses every lookup. Recover with a comparer
that keeps full index keys (same name, so the manifest still matches)
and no filter.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* filer: recover a leveldb store when a compaction finds a corrupted table
When a compaction reads a damaged table block, as a power loss can
leave one, goleveldb latches the corruption error and returns it for
every later write until the database is reopened. A plain reopen loads
the same table and the next compaction fails again, so the store kept
rejecting inserts even across restarts.
Treat a corruption error from a write like a storage error, and reopen
through leveldb.Recover, which rebuilds the table without the damaged
blocks, then retry the write.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* filer: rebuild damaged leveldb tables before Recover
leveldb.Recover's own table rebuild orders internal keys with the user
comparer, so a damaged table holding several versions of one key fails
with "keys are not in increasing order" and the recovery gives up.
Overriding the comparer cannot fix that, because the same comparer
also serves the recovered database's user keys.
Rebuild every table that has corrupted blocks first, using an internal
key comparer and full index keys, then let Recover rebuild the manifest
from tables it no longer needs to rewrite. Seed the test tables from a
journal flush so they keep several versions of a key, and check keys
the write test does not rewrite.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* filer: leave unreadable leveldb tables for Recover to drop
table.NewReader reports a truncated table or a damaged footer through
the reader, not its error, so rebuildDamagedTables never stopped on one,
but it rewrote it as an empty table first. Skip tables with nothing
readable and let Recover drop them, and test an unreadable table next
to a healthy one.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* filer: let StrictRecovery drop damaged leveldb tables
With StrictRecovery set, leveldb.Recover drops a table that has corrupted
blocks. Salvaging the readable blocks first would hide the damage from it,
so skip the rebuild under that option.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* filer: hold the leveldb storage lock while rebuilding tables
The table rebuild ran after the failed open released the storage lock
and before Recover took it, so it could rewrite the tables of a database
another caller had open on the same storage. Take the lock for the
rebuild; Recover takes it again afterwards.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* master: reject an assign on a follower as Unavailable
Assign on a master that is not the raft leader returned
raft.NotLeaderError as a plain error, which reaches gRPC clients as
codes.Unknown. operation.Assign retries only Unavailable,
ResourceExhausted, Canceled and DeadlineExceeded, so every assign made
during a leader election failed at once instead of waiting for the new
leader.
Return it as codes.Unavailable, like the warming-up status in the same
file, so clients retry within their 30 s assign budget. The message is
unchanged, so callers that match "Not current leader" still work.
StreamAssign now answers such a request in-band with the error message
instead of ending the stream, as it already does for warm-up.
TestAssignOnFollowerIsUnavailable covers it.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* pb: keep the cached channel when the peer answers Unavailable with details
gRPC's own Unavailable (dial failure, closing transport) never carries
details, so one that does came from the peer over a working channel.
Closing the shared ClientConn for it cancels every other in-flight RPC
on it with "the client connection is closing".
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* master: tag the follower's assign rejection with a NOT_LEADER ErrorInfo
The detail marks it as the master's answer, so clients retry it
without tearing down their cached channel to that master.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
* test: let the merge-read vacuum outlast the master's write drain
Before compacting a volume the master waits up to 30 s for writes it
assigned to that volume to drain. The test cut each /vol/vacuum call at
30 s and gave the vacuum 60 s, so one or two slow drains failed it.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* test: retry the merge-read vacuum until the master runs one
A 409 means another vacuum holds the master's lock; check the chunks
only after a vacuum this test asked for has finished.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* util/http: probe an unanswering server in the background, not on a read
Once the retry interval passed, the first read put the dead server first
and waited out the full dial timeout before trying a live replica. Keep it
last and dial it from a background probe with a short timeout instead.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* util/http: try every server of a machine that does not answer last
Unreachability was kept per host:port, so a powered-off node running one
volume server per disk cost one stalled read per server before each was
known dead. A dial that times out or finds no route now marks the whole
machine; a refused connection still marks only that port and shows the
machine is up.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* util/http: probe an unanswering server with an HTTP request
A bare TCP connect cleared servers that accept connections but never send
response headers, and could not dial an address without an explicit port.
Probe with a HEAD request through the global client instead, and have the
tests wait for the probe to settle.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* util/http: keep a timed-out port last after a sibling port answers
A dial timeout stored only the machine verdict, so a success on another
port of that machine cleared it and the port that timed out went back to
the front. Record the port as well as the machine.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* util/http: classify refused and unroutable dials with Winsock errnos on Windows
Windows reports these as WSAECONNREFUSED, WSAEHOSTUNREACH and friends,
which errors.Is does not match against the syscall constants. Test the
refusal with a real dial to a closed port.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* util/http: wait for background probes through a WaitGroup in tests
A refused probe can finish before the test reads the claimed record, so
inferring completion from the record's timestamp could wait out the
deadline on a probe that already settled.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* util/http: key machine verdicts apart from hosts without a port
A host without an explicit port is also its machine name, so a sibling
port answering cleared the timed-out host along with the machine verdict.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* util/http: group a bracketed port-less IPv6 host with its machine
net.SplitHostPort rejects "[2001:db8::9]", so the machine key kept the
brackets and missed the verdict stored for "[2001:db8::9]:9501".
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* filer: metrics for metadata subscription delivery (lag, holds, ring fall-offs, claim and watermark ages)
* filer: measure subscribe delivery lag after the real send using the mutation stamp, and record a watermark hold once per hold
* filer: skip empty progress markers when sampling subscribe delivery lag
* filer: assert subscribe delivery lag is sampled only after the stream send succeeds
* grafana: panels for the metadata subscription delivery metrics
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* filer: record a watermark hold when the read passes the held entry, not when the next hold starts
* filer: use a monotonic clock for the subscribe hold duration
---------
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* filer: keep an explicit gRPC port in the embedded gateways' filer address
runFiler handed the embedded S3/WebDAV/IAM/SFTP gateways
NewServerAddress(...).String(), and String() drops the .grpcPort part.
With a non-default -port.grpc the gateways then dialed port+10000 and
did not recognize their own filer in the master's list.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* wdclient: match discovered filers by gRPC address
A filer listed as host:port and the same filer discovered as
host:port.grpcPort were compared as raw strings, so a refresh pruned the
configured filer, added every discovered filer as new in map order, and
reset the current filer to whichever came first; a pushed join added it
a second time. Key filers by their gRPC address: a survivor keeps its
health and current position and takes the discovered address form, and
pushed joins and leaves match the same way.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* security: scoped read-only token for filer metadata subscriptions
A token signed with jwt.filer_signing.subscribe.key names the subscribe
methods and the path prefixes its holder may see. mTLS peers and admin
tokens stay unrestricted. Internal subscribers attach an admin token
when both keys are configured.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* filer: enforce subscription token scope on metadata streams
With jwt.filer_signing.subscribe.key set, SubscribeMetadata and
SubscribeLocalMetadata authorize the caller first. A scoped caller gets
a stream that sends only events whose paths are all under its prefixes,
filters batches, and never carries log file refs.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* filer: send an admin token on internal metadata subscriptions
Peer filers, metadata followers and the lifecycle reader attach an
admin token when the subscribe key is configured, so they keep the full
stream without mTLS.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* shell: add fs.meta.token to mint metadata subscription tokens
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* filer: end scoped metadata subscriptions at token expiry
The scope now carries the token's exp. The scoped stream's context ends
at that time and the call returns Unauthenticated, so an open stream no
longer outlives its token.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* filer.sync: sign metadata subscriptions with the source cluster's key
MetadataFollowOption takes a FilerJwtProvider; filer.sync passes the
one loaded from -a.security or -b.security, so clusters with different
admin keys both accept the subscription.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* shell: report fs.meta.token flag errors
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* filer: give scoped metadata subscribers their own client id
A scoped caller could reuse another subscriber's ClientId with a higher
epoch and replace that registration, disconnecting it. Scoped streams now
register under a server-chosen id that no other subscriber holds.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* security: accept only HS256 metadata subscription tokens
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* security: test HS384 metadata subscription token rejection
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* filer: never let a scoped subscriber replace another registration
The server-chosen id can be taken by another subscriber between choosing
and registering it. Registering scoped streams with the lowest epoch makes
addClient refuse them rather than replace the other registration.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* filer: send position markers for events a scoped token withholds
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* filer: test a scoped stream withholding a whole event batch
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* filer: filter SendMsg on scoped metadata streams
The scoped wrappers only overrode Send, so a SendMsg on the embedded
stream would reach the client unfiltered. Route it through the same
filter and refuse any other message type. Ported from #12790.
Co-authored-by: mindaeon <30555587+mindaeon@users.noreply.github.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* filer: test a scoped subscription through the real subscribe loop
Run SubscribeLocalMetadata with a scoped token over a flushed backlog,
the in-memory tail and live events: only in-scope events arrive, sibling,
"..", encoded and cross-boundary rename paths are dropped, and no log
file references are sent although the client asks for them. Without the
subscribe key every event still arrives. Ported from #12790.
Co-authored-by: mindaeon <30555587+mindaeon@users.noreply.github.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* filer: test that the HTTP API refuses metadata subscription tokens
GET, HEAD, PUT, POST and DELETE all refuse a subscription token, also
when the subscribe key equals the read or write key. Ported from #12790.
Co-authored-by: mindaeon <30555587+mindaeon@users.noreply.github.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: mindaeon <30555587+mindaeon@users.noreply.github.com>
* s3: list each part number once, its latest attempt
Every UploadPart writes its own NNNN_<uuid>.part entry, so a part number
uploaded twice listed twice, superseded ETag included, and a max-parts page
could hold only the stale attempt. ListParts now groups the entries by part
number, reports the attempt sortEntriesByLatestChunk ranks first (the same
order completion uses), and pages over distinct part numbers, reading past a
page boundary that splits one part number's attempts.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3: test ListParts over repeated part attempts
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3: rank an empty part attempt by commit stamp or mtime
An empty UploadPart body stores no chunks, so sortEntriesByLatestChunk gave
it time zero and an empty re-upload always lost to the attempt it replaced.
Attempts now rank by the filer's commit stamp when both carry one, else by
the later of the first chunk's write time and the entry mtime.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3: keep only the newest attempt per part while listing parts
listPartAttempts held every attempt of a part number until the page was
built, so a heavily retried part grew one ListParts request's memory past
what max-parts bounds. It now compares each entry against the newest seen
for its part number with compareLatestPartAttempt, the comparator
sortEntriesByLatestChunk sorts with.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3: order part attempts by commit stamp, then attempt time
Comparing commit stamps only when both attempts carried one made the order
non-transitive across stamped and unstamped attempts, so listing and
completion could pick different attempts depending on input order. The
commit stamp now leads (unstamped entries predate the field) and the
attempt time breaks ties.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3: list part 10000 after the four-digit part numbers
Part names sort as text, so 10000_<uuid>.part lands between 1000.part and
1000_<uuid>.part. A page could end on part 10000 and the next marker then
skipped parts 1000-9999, and a legacy 1000.part could be reported before
the newer attempt of part 1000 was read. List the four-digit part numbers
first and part 10000 in its own prefix listing.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3: test ListParts paging across part 10000
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
test/s3: build s3-fast-list with its data map draining the last page
The data map task reads the channel and then the list-task state. When
the list task sends its last page and completes between the two reads,
the map is dumped without that page and the tool exits 0, so a one-page
prefix comes back empty. Read the state before the channel.
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* telemetry: refresh the server module to the root module's dependency versions
The server module replaces seaweedfs with ../.., so root dependency
bumps raised prometheus/common, prometheus/procfs and x/sys past the
versions its go.mod listed, and go build refused to run.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* ci: run the telemetry integration test on root go.mod changes
The telemetry server module builds against the root module through a
replace directive, so a root dependency bump can leave its go.mod stale
without touching telemetry/.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
The versions chart keeps the 8 versions with the most clusters on the last
day and folds the rest into "other". A new release starts small, so 4.54
(282 clusters) was folded in while 4.53 shrank as clusters upgraded,
which showed as a dip across the named versions. The newest plain release
now always keeps its own band.
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* mount: release the meta cache lock between batches of a rebuild wipe
deleteFolderChildrenForRebuild held mc's write lock while it listed every
cached child of the directory and deleted them. Every metadata read in the
mount (FindEntry, IsNameFresh, isOversized) takes the read lock, and a
waiting writer blocks new readers, so while a large directory was wiped,
lookups and getattrs in unrelated directories queued behind the scan. The
scan gets slower as leveldb accumulates the deletion markers of earlier
wipes of the same directory, which happens quickly once lookups rebuild
expired directories.
Wipe rebuildWipeBatchSize (1024) children per lock hold and drop the lock
between batches. A child is judged pinned under the same lock that deletes
it, so a local-only create is kept whether it was pinned before the wipe
started or between batches. The listing that refills the directory already
ran outside the lock, so batching adds no interleaving the build did not
allow.
Add LevelDBStore.BatchDeleteEntries, the counterpart of BatchInsertEntries.
Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
* mount: bound how long a lookup waits on the directory rebuild it starts
Since #11268 a lookup whose parent's cache expired rebuilds the parent and
every expired ancestor up to the mount root before it answers, and every
lookup beneath them joins the same builds. With a slow listing (a large
directory, a busy filer, or the meta cache lock held by another rebuild) a
plain stat, an ls -l, or the parent touch after mkdir waited for all of it.
Wait at most lookupRebuildTimeout (2s), then fall back to one LookupEntry
RPC. Since #12206 the shared build does not run on the requester's context,
so it keeps going after the waiter gives up and later lookups find the
directory cached. A timed-out wait records a rebuild attempt, so lookups
during the cooldown do not queue on a build that is already known to be
slow.
Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
* mount: wait for the rebuild's listing in TestLookupDoesNotWaitOutSlowRebuild
The build runs on its own goroutine, so the lookup that stopped waiting
for it can return before the build's ListEntries reaches the filer. The
test read `started` without waiting right after the lookup returned, which
relied on an ordering nothing guarantees, and the ListEntries count taken
afterwards assumed the same. Wait for the listing to arrive, bounded by
the test's deadline.
Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
* mount: keep children inserted while a rebuild wipe runs
A deferred create inserts its placeholder before Create registers the
handle that pins it. The single-lock wipe made such an insert wait until
it finished, so the placeholder survived; a batch that runs in that gap
judged it unpinned and dropped it, and the rebuilt listing then omitted
the file until its flush.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* mount: drop a cached path that an event vacates during a rebuild wipe
A build replays only the events later than its listing's snapshot and
leaves the earlier ones to the listing, which starts after the wipe. With
the wipe in batches a file's whole life fits inside it: created, flushed,
then unlinked or renamed away. Both events are earlier than the snapshot
and are skipped, the listing has no such file, and the placeholder the
wipe kept stays in the published directory until the next rebuild.
While a directory's wipe runs, an event buffered for it now takes the
path it vacates out of the store at once, fenced by versions like any
other event. After the wipe the store holds what the listing put there,
and a removal earlier than the snapshot may still arrive, so from then on
the event is left to the replay as before.
Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
* mount: check which file a removal took away during a rebuild wipe
Two things the removal applied during a rebuild wipe got wrong.
A placeholder is unversioned, so the version gate lets every removal
through. One that arrived late, for an earlier file of the same name,
took out the placeholder of a file created since. The entry the event
took away is now compared with the cached one, by inode or, with no
inode to compare, by creation time. An event that only names the file
cannot tell and is taken at its word.
The entry was deleted without OP=MV, so taking one name of a hard-linked
file out of the cache also took one off the count its other names share.
A replayed event that vacates a path already deletes with OP=MV.
Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
* mount: keep a removal the rebuild wipe declined out of the replay
The wipe declines a removal when what is cached there is newer than the
event or is another file than the one it took away. The event stayed
buffered anyway. An unversioned listing clears the children's version
records and replays every buffered event, so the declined removal then
deleted the entry the wipe kept and the directory was published without
it.
A declined removal now buffers only the name it puts in place, so a
rename still lands on replay.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* mount: check a declined removal again when the build replays it
The previous commit dropped the removal half of an event the wipe
declined, judged by the file cached when the event arrived. Earlier
buffered events can replace that file during the replay: a listing
attempt that failed leaves its events buffered, so the retry's wipe may
see the first attempt's file while the replay puts a later one in place
first. The dropped delete then left that later file published.
The wipe now records which removals it declined, and the replay checks
each against the entry it finds at that point, replaying only the name
the event puts in place when that entry is still another file. A
removal declined by version is replayed as before.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* mount: keep a declined removal's invalidation off the placeholder's handle
After the replay the build invalidates every buffered event again,
raised to the snapshot, and it still sent the original of a removal the
replay rejected. A late delete of an earlier file then reached the
handle of the placeholder at that name: the placeholder is unversioned,
so the raised timestamp got past it and the handle was marked deleted.
The replay's decision on a declined removal now replaces the event in
the buffer, and the invalidations use what was replayed. It is made
before the snapshot check, so a removal the listing covered is judged
the same way.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* mount: keep an event's signatures on the halves a rename is split into
A declined removal replays as its rename's create half, and a rename across
a building directory is split in two. Both dropped the event's filer
signatures, so a handle stamped by that filer no longer fenced the replayed
invalidation by timestamp and could take older listed contents over newer
ones it had read.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* mount: drain invalidations around the declined-removal build check
Invalidations run on a background worker, so the baseline could miss the
removal's own and the final read could run before the build's arrived.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: MapleWhiteZZZ <18653045+MapleWhiteZZZ@users.noreply.github.com>
Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
* volume: fsync a batch delete that asks for it
BatchDeleteRequest gets an fsync field, and the Go and Rust volume
servers pass it to the needle delete, which already orders the
tombstone, .dat sync, publish and .idx sync for a durable delete.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* operation: pass an fsync flag to batch deletes
DeleteFileIdsWithLookupVolumeId and DeleteFileIdsAtOneVolumeServer take
an fsync argument and send it in the BatchDeleteRequest. Every caller
passes false for now.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* filer: fsync chunk deletes under an fsync path rule
DeleteChunks passes the path rule's fsync flag to the deletion queue.
The flag is kept with the pending record until the delete is
confirmed, so retries and re-queues keep it, and the batch and retry
loops send marked ids in a durable BatchDelete.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* filer: keep fsync marks in the deletion ledger
Marked ids go into the trailing parts of a manifest ledger, counted by
fsyncParts. Older filers read those parts as plain ids and still delete
them; a reload or a claimed foreign ledger restores the marks. A ledger
without marks keeps the single array form.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* filer: record a queued deletion before the delete loop can see it
queueDeletions published the ids to the hot queue before taking the
ledger lock, so a concurrent flush could delete an id before its fsync
mark landed and then forget the mark with the record.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* filer: fsync a folder delete's chunks by each child's path rule
A recursive delete collected its file children's chunks and resolved
the rule once for the folder, which misses a trailing-slash rule on
the folder itself and file-prefix rules. Each file child's own rule now
decides fsync; the folder rule still decides chunk deletion.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* filer: keep a newer fsync request when a plain delete finishes
An id re-queued with fsync while a plain delete of it was in flight
lost its mark when that delete succeeded, so the queued copy also ran
without fsync. A plain attempt now leaves a marked record for its
queued copy; a durable delete of an already-deleted needle still syncs
the tombstone.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* filer: delete a folder's plain child chunks without fsync
Children whose own rule turns fsync off were still deleted durably when
the folder's rule asked for fsync. Only the children marked by their
own rule now go out with fsync.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* filer: retry a durable chunk delete the volume server failed
A volume server rolls back a durable delete it cannot sync and answers
with a 5xx carrying the I/O error text, which the classifier took as
permanent, so the record and its fsync mark were dropped while the
needle stayed live. A 5xx answer to a durable delete is now retryable,
within the usual attempt limit.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* s3api: build request condition keys in one helper
Split the keys ExtractConditionValuesFromRequest reads from the request
itself into RequestConditionValues, keeping aws:SourceIp and the
bucket-listing keys in the method.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3api: evaluate request condition keys for STS sessions
The IAM integration context for STS and OIDC sessions carried only the
source IP, legacy user agent/referer keys and claims, so role and session
policy conditions on s3:x-amz-*, aws:SecureTransport, aws:CurrentTime,
s3:RequestObjectTag/* and the other request keys never matched: a Deny
on them never fired, and a negated Deny always did. Add the same request
keys the static identity path evaluates.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3api: report SigV4 presigned requests as REST-QUERY-STRING
s3:authType recognized only SigV2 query authentication, so a policy
condition on REST-QUERY-STRING never matched a SigV4 presigned URL.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* ci: cache the Snowflake suite's Maven dependencies under their own key
setup-java's maven cache key hashes the repository pom.xml files, which
other Java workflows share. The Snowflake job restored that cache, which
lacks the suite's plugins and dependencies, and never saved its own because
the key already existed. Every run downloaded the whole suite dependency
tree from Maven Central and failed when Central answered 429.
Key the cache on run.sh, which pins the suite revision.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* ci: fall back to Google's Maven Central mirror in the Snowflake job
A cold cache still resolves about 36 MB of plugins and dependencies, and
Maven gives up after three 429 retries within 30 s. Add the mirror as a
settings-profile repository; Maven tries it first and moves on to Central
for any artifact the mirror fails to serve.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* filer: mint read_auth for the data chunks behind manifest chunks
* filer: cover every chunk behind manifests in read_auth (data + nested), minted after resolution, and skip when read signing is off
* filer: pass the master client as manifest cache invalidator, and move the read_auth helpers below populateBatchReadAuth for the enterprise merge
* testutil: fetch pinned Maven Central jars with a mirror fallback
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* test: mount pinned Iceberg jars into the Spark catalog containers
Each test started a new Spark container that resolved the Iceberg runtime
and AWS bundle through Ivy from Maven Central, so a run downloaded them six
times and a single refused request left Spark retrying until the package
timed out.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* ci: cache the Iceberg jars for the Spark catalog tests
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* testutil: test the verified jar download
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* test: bound Maven jar downloads by one deadline shared across retries
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* test: give each Maven mirror attempt its own timeout within the download budget
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* s3api: accept an empty ListObjectsV2 continuation-token
An empty continuation-token lists from the start and is echoed back
empty, as S3 does; rejecting it broke s3-tests
test_bucket_listv2_continuationtoken_empty.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3api: leave the ListObjects V1 Prefix unencoded under encoding-type=url
S3 encodes the V1 Delimiter, markers and keys but echoes Prefix as sent,
and botocore, which always asks for url encoding, decodes the V1 Prefix
only for V2. Encoding it broke s3-tests test_bucket_list_prefix_unreadable.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3api: use a codespell-clean key in the versions encoding test
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3api: URL-encode a V1 Prefix that XML cannot carry
Leaving the V1 Prefix raw under encoding-type=url turned a control
character such as \x01 into U+FFFD in the response. Such a prefix is
URL-encoded again; prefixes that are valid XML text, including '\n',
stay as sent.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* master: export writable volumes per disk for each volume layout
The layout gauges only report how many volumes of a collection take
writes, not where they sit, so two writable volumes sharing one disk
cannot be seen or alerted on. The leader now exports
writable_volumes_per_disk with the layout labels plus data center, rack,
node and disk id, counting the replicas of the layout's writable volumes
on each disk (state=writable) and those of them that are crowded
(state=crowded). Series for disks that no longer hold a writable volume of
the layout, and for layouts that are gone, are deleted.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* master: reset the leader-only layout gauges on followers
The layout writable, crowded, under-replicated and per-disk gauges are only
set by the leader, so a master that lost leadership kept exporting the
values of its last term indefinitely, and dashboards that aggregate over
masters could show numbers days old. A master that is not the leader now
clears them on each periodic pass.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* grafana: chart disks holding several writable volumes of a collection
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* grafana: keep disks of different clusters apart in the per-disk panel
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* s3api: reject an empty ListObjectsV2 continuation-token
AWS answers continuation-token= with 400 InvalidArgument ("The continuation
token provided is incorrect"); SeaweedFS listed from the start of the bucket.
No token this server issues is empty, and raw-key and base64: tokens keep
resuming as before.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* s3api: reject an encoding-type other than url on listings
AWS answers any encoding-type other than url with 400 InvalidArgument
("Invalid Encoding Method specified in Request") on ListObjects,
ListObjectsV2, ListObjectVersions, ListMultipartUploads and ListParts;
SeaweedFS ignored the value and listed unencoded keys.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* s3api: apply encoding-type=url to ListObjectVersions
The handler accepted encoding-type=url but returned raw keys. It now
URL-encodes Key in Version and DeleteMarker entries, CommonPrefixes, Prefix,
Delimiter, KeyMarker and NextKeyMarker with the urlPathEscape form ListObjects
uses, and echoes EncodingType.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* s3api: URL-encode Prefix and Delimiter in ListObjects responses
Under encoding-type=url, ListObjects and ListObjectsV2 encoded keys,
CommonPrefixes, markers and StartAfter but echoed Prefix and Delimiter raw,
so a client decoding them once corrupted a prefix holding '%' or '+'.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* volume growth: spread writable volumes of a layout over servers and disks
Volume growth picked every server, and the volume server picked every disk,
independently: the server weighted by free slots, the disk by the lowest
total volume count. A grow request therefore stacks several writable volumes
of one collection on some disks while most disks get none (a Poisson spread:
with 160 replicas on 479 disks about 20 disks get two or three), and their
writes share those disks.
Data centers and racks are still picked weighted by free slots. Inside a
rack the server is now one of those holding the fewest writable volumes of
the layout per disk, weighted by free slots among them; this covers the main
server, same-rack replicas and replicas walked into other racks and data
centers. On the volume server the disk is the one with the fewest writable
volumes of the collection, then the lowest volume count as before.
A grow request registers each volume before placing the next, so a request
smaller than the number of disks lands on distinct disks.
Fixes#12746
* volume growth: scan the layout load once per grow request
Placing each volume rescanned all writable volumes of the layout under its
read lock. RecordAssign takes that lock exclusively on every assign, so with
thousands of writables (0.8 ms for 10000) every volume of a grow request
stalled assigns once more.
GrowByCountAndType now scans once and findAndGrow adds every volume it
creates to the counts. Per volume, placement costs about 130-160 us on 480
servers whatever the number of writables, against 120 us before the spread
change; the single scan is 0.17 ms for 2000 writables and 0.8 ms for 10000.
* volume growth: address review of the spread placement
- Count only disks that report volume slots in a server's load. An
unavailable disk, or one low on space with no volumes, is still listed in
DiskTags with no slots; counting it understated the load of a server with
failed disks and piled volumes onto its remaining ones.
- On the volume server, judge a volume full by its .dat size, which the
master compares with the limit, not by ContentSize.
- A server that refuses an allocation ranks last for the rest of the grow
request, and the request retries the volume up to 3 times. The least
loaded server is picked first deterministically, so without this one server
that cannot fit a preallocated volume stopped the layout from growing.
- Scan the layout load after warmup, so a grow request issued right after a
leader change sees the writable volumes the returning servers report.
* volume growth: keep the request-local placement load ahead of the request fields
The JSON request fields stay together at the end of VolumeGrowOption, and
the comment now says the load lives for the whole grow request.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* volume(rs): place a new volume on the disk with the fewest writable volumes of its collection
Store::add_volume picked the disk with the most free slots, so the Rust
volume server ignored the collection spread rule the Go server now applies.
find_free_location ranks disks by the writable volumes of the collection
first (not read-only, .dat below the volume size limit), then by free slots.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* volume growth: leave a refusing server out of every placement tier
A server that refused an allocation only ranked last among the servers of
its rack, so the weighted data center and rack picks could still choose its
rack on every retry while other racks had room. It now offers no free slots
for the rest of the grow request, which takes its slots out of the rack and
data center weights too. If only refusing servers are left, they are tried
again.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* volume growth: require the full same-rack count only in the main rack
The data center check asked every rack for SameRackCount+1 free servers,
but only the main rack holds the same-rack copies; each other rack takes one.
With a refused server left out of its rack, the check rejected a feasible
placement, the retry put the refused server back, and it was picked again.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Konstantin Lebedev <whitefox@mayflower.work>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* volume: fsync a needle delete that asks for it
A delete now takes fsync like a write: the tombstone is synced before the
needle map drops the needle, and the .idx row before the delete is
acknowledged. A failed .dat sync takes the tombstone back off the .dat.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* volume: honor fsync=true on DELETE and forward it to replicas
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* volume(rs): fsync a needle delete that asks for it
Mirrors the Go volume server: DELETE with fsync=true flushes the tombstone
before the index drops the needle and the .idx before the ack, and a failed
.dat flush takes the tombstone back off the .dat.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* volume: let a durable delete through on a full volume
A queued delete hit the batch worker's volume size check, which a
tombstone append is exempt from inline, so fsync=true deletes failed once
the volume reached its hard limit.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* volume: sync a durable delete of an already-deleted needle
An earlier delete without fsync may have dropped the needle with its
tombstone still unsynced, so the inline path syncs before acking anyway,
as the batch worker already does.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* volume: refuse a batch request that races the worker stopping
stopWorker can close the batch channel between startWorker handing it out
and the send, which panicked. The request now falls back to the inline path.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* volume(rs): sync a durable delete of an already-deleted needle
Mirrors the Go volume server: an earlier delete without fsync may have
dropped the needle with its tombstone still unsynced.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* volume(rs): run a durable delete on a blocking thread
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* volume: keep syncDelete's signature for existing callers
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* volume: refuse a write a delete cancelled before it appends
A write cancelled while still queued behind the volume locks used to
append, publish and then tombstone itself without a sync, after its
delete may already have been acknowledged as durable. A crash could
keep the row and lose the tombstone. Refuse it before the append.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* volume(rs): refuse a write a delete cancelled before it appends
Mirrors the Go change: a write cancelled while still queued behind the
volume locks no longer appends, publishes and tombstones itself without
a sync after its delete may already be durable.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* volume(rs): drop the write guards for a durable delete's flushes
The durable delete held the write lease and the io write guard across
both fsyncs, so reads of the volume waited on them. Append and publish
the tombstone under the guards and flush with only the io read side, as
write_synced_run and the Go path already do. do_delete_request keeps its
signature.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* volume: stop taking writes when a durable delete fails to index
The needle map drops the needle before it appends the .idx row, so after
a failed append a durable retry found nothing to delete and acked, and a
later write could bury the unindexed tombstone so a reload brought the
needle back. Fence the volume the way a failed durable write does.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* volume(rs): stop taking writes when a durable delete fails to index
Rust counterpart of the Go change: the needle map drops the needle
before it appends the .idx row, so a failed append left a durable retry
nothing to delete, and a later write could bury the unindexed tombstone.
publish_delete_request now fences the volume for a durable delete, as
publish_write does.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* volume: stop taking writes when any delete fails to index
A non-durable delete whose .idx append failed left the volume writable with
the needle already dropped from the map, so a later durable retry found
nothing to delete and acknowledged without the missing row.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* volume(rs): stop taking writes when any delete fails to index
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* volume: take the fsync flag on the existing delete functions
DeleteVolumeNeedle, deleteNeedle2 and syncDelete take fsync instead of
gaining *WithFsync variants.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* volume(rs): take the fsync flag on the existing delete functions
delete_volume_needle and delete_needle take fsync instead of gaining
*_with_fsync variants; the durable path and the deletable checks fold
back into delete_needle, and the cancelled-write check mirrors Go's
isCancelledWrite.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* test(s3): s3-listing-study integration suite (WIP)
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* test(s3): run the s3-listing-study tools against SeaweedFS (WIP)
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* test(s3): s3-listing-study deleted-prefix probe, fresh data dir per run
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* test(s3): listing study ports keep clear of every gRPC companion
free_ports.py compared only base ports, so a base port could land on
another server's base+10000 gRPC listener.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* test(s3): listing study ports stay below the ephemeral range
Base ports of 20000-55000 put both them and their gRPC companions inside
32768-60999, so an outgoing connection made while weed server starts could
take a port before the server that was given it binds.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* test(s3): listing study checks the order inside each listing page
parse_page sorted Contents and CommonPrefixes together before the probes
compared them, so keys returned out of order within one page passed. Check
each list's byte order first; the versions pager does the same.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* test(s3): listing study applies --only before running a probe
The filter was applied to the probe results, so a run limited to one tool
still sent every reference listing request and the deleted-prefix check.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* test(s3): listing study requires CommonPrefixes from shallow modes
The CommonPrefixes comparison ran only when a tool returned at least one
prefix, so a delimiter listing of the canary root that came back empty
passed. Compare them for every shallow mode except the two whose
normalizer keeps keys only.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* s3api: reject unknown elements in PutBucketNotificationConfiguration
xml.Unmarshal skips children that match no field, so a configuration
whose only child was an unmapped element parsed as empty and returned
success while nothing was stored.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* test/s3/tigris_compat: fail test-with-server when the server does not stop
A passing suite returned success even when stop-server gave up with the
server still running.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* test/s3: run tigrisdata-community s3-api-compat-tests in CI
Runs the suite, pinned to a commit, against weed mini through a small
provider patch, and fails on any test that is not in
expected_failures.txt or on a listed test that starts passing.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* test/s3/tigris_compat: prepare the suite again when the patch changes
The checkout was reused whenever it sat at the pinned commit, so an
edited seaweedfs-provider.patch, or a checkout whose patch failed to
apply, kept being built as it was. Record the commit and the patch's
hash once the patch applies, and prepare the checkout again when either
differs.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* test/s3/tigris_compat: keep the pid file of a server already running
start-server removed weed-server.pid before it found the port taken, so
a second start lost the pid of the server holding it and stop-server
could no longer stop that one. Remove the file only once the port is
free, and have test-with-server stop only a server it started.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* test/s3/tigris_compat: bound the S3 port probes
A listener that accepts but never answers held curl, and with it the
readiness loop and the cleanup after it, for good. Give each probe two
seconds.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* test/s3/tigris_compat: expect a listed test to end with its listed status
A listed test passed the check with any non-pass status, so a setup error
that skipped the call under test (RestoreObject's PutObject, say) went
unnoticed. Entries now expect fail unless they name another status; only
GetBucketAccelerateConfiguration, whose setup PUT is unsupported, expects
error.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* test/s3/tigris_compat: refuse to start while the recorded server still runs
A second start-server before the first server answered on the S3 port
passed the port probe and dropped the pid file of the live server, so
stop-server could no longer stop it.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* test/s3/tigris_compat: recognize the test server by its command line
start-server and stop-server took any process named weed at the recorded
pid for the test server, so a weed process that reused the pid would be
reported as running and killed by stop-server. Match the command line
start-server launches instead.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* test/s3/tigris_compat: recognize the test server by its start time
A weed mini started from another checkout has the same command line, so
a pid it reused still matched. Record the process start time next to the
pid at launch and require both before treating the process as the test
server.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* test/s3/tigris_compat: PutObjectTagging_VersionId passes now
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* test/s3/tigris_compat: keep the server records until the server exits
stop-server waited a fixed two seconds after SIGTERM and then removed
weed-server.pid and weed-server.started even if weed was still shutting
down, so a later stop-server could not find it and clean could delete
data it still used. Wait up to 30 seconds for the recorded server to exit
and keep the records, failing, while it still runs.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* test/s3/tigris_compat: start the server on an empty data directory
A data directory left by an earlier run on other ports made the master
freeze volume growth (topology id mismatch), and every PutObject failed.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* test/s3: run tigrisdata-community s3-api-compat-tests in CI
Runs the suite, pinned to a commit, against weed mini through a small
provider patch, and fails on any test that is not in
expected_failures.txt or on a listed test that starts passing.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* test/s3/tigris_compat: prepare the suite again when the patch changes
The checkout was reused whenever it sat at the pinned commit, so an
edited seaweedfs-provider.patch, or a checkout whose patch failed to
apply, kept being built as it was. Record the commit and the patch's
hash once the patch applies, and prepare the checkout again when either
differs.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* test/s3/tigris_compat: keep the pid file of a server already running
start-server removed weed-server.pid before it found the port taken, so
a second start lost the pid of the server holding it and stop-server
could no longer stop that one. Remove the file only once the port is
free, and have test-with-server stop only a server it started.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* test/s3/tigris_compat: bound the S3 port probes
A listener that accepts but never answers held curl, and with it the
readiness loop and the cleanup after it, for good. Give each probe two
seconds.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3api: accept PutBucketAccelerateConfiguration with Status=Suspended
GetBucketAccelerateConfiguration always reports Suspended, but the PUT
answered 501 for every request. Suspended is now accepted, Enabled
stays NotImplemented, and the accelerate subresource resolves to
s3:GetAccelerateConfiguration / s3:PutAccelerateConfiguration.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* s3api: accept an empty PutBucketNotificationConfiguration
GetBucketNotificationConfiguration always reports an empty
configuration, but the PUT answered 501, including for the empty
configuration clients send to clear notifications. An empty one is now
accepted; a topic, queue, Lambda or EventBridge destination stays
NotImplemented.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* s3api: reject accelerate and notification bodies with the wrong root
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* test/s3/tigris_compat: expect a listed test to end with its listed status
A listed test passed the check with any non-pass status, so a setup error
that skipped the call under test (RestoreObject's PutObject, say) went
unnoticed. Entries now expect fail unless they name another status; only
GetBucketAccelerateConfiguration, whose setup PUT is unsupported, expects
error.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* test/s3/tigris_compat: refuse to start while the recorded server still runs
A second start-server before the first server answered on the S3 port
passed the port probe and dropped the pid file of the live server, so
stop-server could no longer stop it.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* test/s3/tigris_compat: recognize the test server by its command line
start-server and stop-server took any process named weed at the recorded
pid for the test server, so a weed process that reused the pid would be
reported as running and killed by stop-server. Match the command line
start-server launches instead.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* test/s3/tigris_compat: recognize the test server by its start time
A weed mini started from another checkout has the same command line, so
a pid it reused still matched. Record the process start time next to the
pid at launch and require both before treating the process as the test
server.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* test/s3/tigris_compat: PutObjectTagging_VersionId passes now
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* test/s3/tigris_compat: keep the server records until the server exits
stop-server waited a fixed two seconds after SIGTERM and then removed
weed-server.pid and weed-server.started even if weed was still shutting
down, so a later stop-server could not find it and clean could delete
data it still used. Wait up to 30 seconds for the recorded server to exit
and keep the records, failing, while it still runs.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* test/s3/tigris_compat: start the server on an empty data directory
A data directory left by an earlier run on other ports made the master
freeze volume growth (topology id mismatch), and every PutObject failed.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* s3api: return x-amz-version-id from object tagging on versioned buckets
PutObjectTagging, GetObjectTagging and DeleteObjectTagging now report
the version they acted on, "null" for an object written before
versioning was enabled.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* s3api: set the tagging x-amz-version-id only on success
The header was set before the tag-conditioned policy recheck, so a
denied request carried the version id on its AccessDenied response.
Also test the "null" version and require distinct version ids.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* test/s3: run tigrisdata-community s3-api-compat-tests in CI
Runs the suite, pinned to a commit, against weed mini through a small
provider patch, and fails on any test that is not in
expected_failures.txt or on a listed test that starts passing.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* s3api: report no ownership controls after DeleteBucketOwnershipControls
GetBucketOwnershipControls reports BucketOwnerEnforced for a bucket that
never stored a setting, so it kept reporting it after the setting was
deleted. The delete now records that it happened, and GET answers
OwnershipControlsNotFoundError until ownership is put again. How
objects are written after the delete is unchanged.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* s3api: answer DeleteBucketOwnershipControls with 204 No Content
AWS answers a successful DeleteBucketOwnershipControls with 204 and no
body; this answered 200 with an empty OwnershipControls document.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* lance: per-table cleanup retention and on-demand purge of deleted rows
A vector deleted from an index stayed in storage until compaction
rewrote its fragment (only above 10% deletions) and the cleanup job's
168 h retention, set once per worker, aged out every version that
referenced it.
Let a table override retain_hours and min_versions in its Lance config
(seaweedfs.cleanup.retain_hours, seaweedfs.cleanup.min_versions), and
let it request a purge (seaweedfs.purge.requested). The cleanup job
proposes a table with a pending purge regardless of the version floor
and then rewrites every fragment with deletions, retrains the indices
(a remapped IVF_FLAT index still holds the deleted vector), keeps only
the latest version, and records the request as served. A version pinned
by a tag fails the purge and leaves the request pending.
The test scans every file of a local table for a deleted vector's key,
metadata and values, and finds none after the purge.
* lance: refuse to purge a sharded vector index
A retrain builds every index name over the whole table, so the purge
turned shards of one vector index (one name per fragment set) into
names that each covered every fragment, indexing every row once per
shard. Check before compacting and fail with the request left pending,
and retrain only the vector indices.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* lance: rebuild scalar indices in a purge
Compaction only remaps scalar indices, and a retrain leaves them alone,
so a purge reported success while a bitmap index still listed the
deleted row's value, and under stable row ids a BTree index still held
the deleted key. Build each scalar index again from the remaining rows
with the parameters it was built with.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* lance: purge through the cleanup job's retention policy
The purge built its own cleanup policy, so a deployment that has the
cleanup job keep more than the floor and the window (versions an
unleased reader may still be on) would have lost them to a purge. Clean
up through retention_policy, the one the job itself uses, with the
latest version kept.
Mark the request served only once the oldest version left is one the
rebuild wrote, and record the rebuild so a purge that has to wait for
its cleanup (a tag, or that policy) retries only the cleanup instead of
compacting and retraining the indices on every pass.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* vector: lease the version every list page reads
A list page read its version without a lease unless it was a segment
that read data and metadata twice. A purge removes every version but
the latest, and only a lease tag stops it, so a listing that opened the
version before the purge's compaction could lose its files mid-read.
Pin it as get_vectors and query_vectors do.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* lance: give the cleanup job a build's execution time
A purge compacts the table and retrains its indices inside a cleanup
job, but the job declared no runtime defaults, so the admin held it to
its 90 s default and could expire a purge mid-rebuild. Declare the same
budget the indices job uses for a retrain.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* mount: drop a peer-unlinked name only while its inode keeps another
Move the survivor check from followSurvivingLink into InodeToPath.RemoveAlias
so the check and the removal happen under one table lock.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* mount: follow a peer unlink to the surviving hard link for closed inodes
A peer deleting one name of a hard link only moved an open handle to the
surviving name. A closed inode kept the deleted name as its primary path, so
the next open by the kernel's nodeid failed with ENOENT.
Co-authored-by: skymazer <54621405+skymazer@users.noreply.github.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* mount: share the cached-entry freshness check of peer invalidations
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* mount: keep a name a local rename reclaimed before the peer unlink arrived
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* mount: keep a name a local rename or link gave the inode after an older peer delete
The cached-entry check only covers a cached parent. InodeToPath now records,
per hard-linked inode, the newest name a local rename or link gave it with
the filer stamp, and the closed-inode peer-delete path keeps that name when
the delete is older. The record lives in a side map dropped on Forget, so
InodeEntry keeps its size class.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* filer: sign rename stream responses with the filer signature
A rename response carries the stamp of the filer that applied it, so the mount
can tell which clock the stamp belongs to, as it does for create acks.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* mount: order a peer delete against a local rename or link only on one filer clock
The stamp of a local rename or link comes from the filer that applied it, and
a delete logged by another filer runs on a different clock. Keep the name only
when both carry the same nonzero filer signature; otherwise drop it as before.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* mount: drop the local-name stamp fence for uncached parents
Ordering a delayed peer delete against a local rename by filer stamp needs a
record per alias, a shared filer clock, and the open-handle path to follow;
that is a larger change than this fix. Back to the cached-entry check, which
covers a cached parent.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: skymazer <54621405+skymazer@users.noreply.github.com>
* mount: report opens for reading with TouchAccessTime (-accessReport)
An open answered from the metadata cache never reaches the filer, so a
filer that tracks access cannot see that a cached file is still being
read. With -accessReport, the mount sends one TouchAccessTime per open
for reading, with the open time, whether the open was answered from the
cache or not. Write-only opens are not reported.
Reports go out in the background and never hold up the open: a report
that does not fit in the queue is dropped, and drops and failures are
logged once a minute. A filer that answers Unimplemented pauses
reporting for 10 minutes.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* mount: address review of the access report
- Drop reports still queued when reporting pauses; the filer that answered
Unimplemented would answer them the same way.
- Bound each filer attempt to 2s, so a filer that accepts the connection
but never answers leaves time for the next one, and keep using the filer
that answered.
- Fail over only when a filer cannot be reached. Any answer, Unimplemented
included, ends the walk.
- Report an O_CREAT open of an existing file with read access, which comes
through Create instead of Open. A new file has nothing to read and is not
reported.
- Stop the reporter on unmount: stop ends the workers and cancels the
report in flight.
- Say in the flag help that it has no effect unless the filer implements
TouchAccessTime.
- Tests no longer wait a fixed time to see that nothing else was sent; they
look at the queue of a reporter that is not started.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* mount: fail an access report over only when the filer did not answer
A status the filer sent ends the walk even when its message contains a
word the transient classifier matches, so an Unimplemented answer still
pauses reporting and a rejected request is not resent elsewhere.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* mount: keep a filer selection made while an access report failed over
A report that reached a later filer moves the mount there only if the
selection it started from is still current, so a slower report cannot
undo another call's failover.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* mount: share an access report's time with the filers not yet tried
Each attempt is capped by an equal share of the report time left, so a
row of stalled filers cannot use up the report before a healthy one later
in the list is tried.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* mount: report the path an open for reading came through
A create that opens an existing hard link reports the name it was
given rather than the inode's first known name. Open reads the inode's
path from the inode table rather than the handle's saved path, which
other handle operations may rewrite concurrently.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* mount: check the access report stop and failover tests without races
The post-stop report is checked on a reporter without workers, so no
worker can take an accepted report before the check, and the failover
test asserts that the stalled filer was tried first.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* pb: keep the shared channel when the peer answers with its own status
gRPC never raises InvalidArgument, NotFound, AlreadyExists,
FailedPrecondition, OutOfRange or DataLoss itself, so such a status is
the peer's answer. It fell through to the string matching, and a message
naming a path like /transport closed the cached channel and cancelled
every other call on it.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* mount: say which filer selection an access report keeps
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* mount: bound the in-flight call in the rejected-report channel test
The call held in flight went through touchAccessTime, whose 2 s attempt
cap could expire on a slow runner and fail the test on a healthy channel.
It now uses callFiler with the test deadline. The wait for it to reach
the filer also ends if it returns early or the deadline passes, and the
blocked handler is released on every exit path.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* mount: report each file at most once per -accessReport.interval
Every read open used to send a TouchAccessTime, adding a filer metadata
call per open. A path reported within the interval (default 1h) is now
skipped; the last report times live in a bounded LRU. 0 keeps reporting
every open.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* mount: report an open again once the file was modified since its last report
Within -accessReport.interval a read open was skipped even after a write,
so the filer kept an atime older than the mtime until the interval ran
out. relatime records that first read after a write; do the same by
comparing the entry's mtime with the last report.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* mount: keep the report time apart from the mtime it saw
Recording max(now, mtime) as the last report let a file dated a day ahead
suppress reports for a day plus the interval. Record when the report was
sent and the mtime the entry had, and treat any change of that mtime as a
modification, so neither clock can stretch or shorten the interval.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: MapleWhiteZZZ <18653045+MapleWhiteZZZ@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
* shell: volume.check.disk skips replicas whose live counts match
FileCount counts every write and DeleteCount every delete or overwrite
until a vacuum drops them, so a replica vacuumed while its peer was not
keeps a different FileCount forever and was re-checked on every run
although both hold the same live needles. Only treat counts as differing
when the live counts (FileCount - DeleteCount) differ too.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* shell: volume.check.disk skips equal live counts only across compaction revisions
Equal live counts at the same compaction revision mean a missed write
and a missed delete, not a vacuum-state difference, so keep checking
those pairs.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* shell: volume.check.disk real-time recheck ignores heartbeat revisions
VolumeStatus counts are newer than the heartbeat's compaction revisions,
so a vacuum in between could make a pair at the same revision look
vacuum-skewed. Apply the old count rules there.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* shell: volume.check.disk skips on live counts only when no repair could follow
Equal live counts across compaction revisions can still hide a missed
write and a missed delete. Keep checking such pairs when -syncDeleted
could propagate the delete, or when -resurrectMissingNeedles could copy
a needle into a never-vacuumed replica.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* s3vectors: enforce AWS metadata, key and filter limits
PutVectors accepted vector metadata of any size and key count, bounded
only by the request body limit, and keys of any length; QueryVectors
accepted filters of any size. Apply the AWS S3 Vectors limits
(https://docs.aws.amazon.com/AmazonS3/latest/userguide/s3-vectors-limitations.html),
taking the ENHANCED index values where the modes differ, since this
implementation prefilters and takes no indexMode:
- 100 KB total metadata per vector, which also bounds a single
non-filterable field;
- 2 KB filterable metadata per vector;
- 50 metadata keys per vector;
- 1024-character vector keys;
- at most 10 nonFilterableMetadataKeys of 1 to 63 characters at
CreateIndex;
- 100 filter constraints per QueryVectors request.
A violation is a ValidationException before the engine is called.
* s3vectors: accept metadata whose JSON is exactly at the byte limits
metadataEntrySize counts a trailing comma for every member, so starting
from both braces overcounted by one and rejected metadata that encodes to
exactly 2 KB filterable or 100 KB total.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3vectors: rewrap the metadata limits header comment
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* kms/openbao: refresh the AppRole token before it expires
The provider logged in with AppRole once, when it was created, and
never renewed the token. After the token's TTL every SSE-KMS upload,
and every read of an SSE-KMS object whose data key was not cached,
failed with "permission denied" (HTTP 500 to the client) until the S3
server was restarted.
With AppRole authentication the provider now logs in again in the
background once two thirds of the token's TTL have passed, and retries
a failed login every 5 s. Each login is bounded by the client's
timeout, or by 30 s when request_timeout is not positive. Close stops
the refresh. A static token, or an AppRole token without a TTL, is left
as it is.
* kms: keep the OpenBao token refresh across configuration reloads
- AddKMSProvider and InitializeGlobalKMS create a provider instance of
their own with the new NewProvider instead of the registry's cached
one: a reload closed the cached instance and then got it back, which
now stopped its token refresh. The new instance is created before the
old one is closed, so a configuration that fails leaves the current
provider working
- openbao: time each refresh from the start of the login that issued
the token, since its TTL runs from then rather than from the response
- tests: a reload gives a new instance and closes the old one, a failed
reload keeps the old one; the refresh survives a reload; a slow login
response
* kms: create a replacement provider outside the manager lock
AddKMSProvider ran the provider factory, which for OpenBao is a login
that may wait for its timeout, while holding the manager lock, so every
lookup, also of unrelated providers, waited for it. The new provider is
now created before the lock is taken; the lock covers only the swap, and
the replaced provider is closed after it. Test: a lookup while another
provider is being created.
PutObject rejected a Cache-Control value it could not parse with
InvalidDigest, whose message is about Content-MD5, even when the request
sent no Content-MD5. CreateMultipartUpload already answers the same parse
error with InvalidRequest; PutObject now does too.
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* operation: reassign an upload after a proven-expired volume JWT
* seaweed-volume: mark an expired write JWT refused before append
Mirror the Go volume server: a multipart POST whose write JWT is signed
with the current key, names this fid and fails only on exp gets
X-Seaweed-Write-Outcome: expired-jwt-before-append on its 401, so the
uploader reassigns instead of failing. Both checks use one guard
snapshot, so a key rotation between them cannot drop the marker.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* operation: keep same-FID retry for filer-proxy expiries, snapshot the write signing key, and make the RetryForever budget test fail fast
* operation: re-mint the filer credential before a proxy same-FID expiry retry
* operation: make a proxy marked-expiry terminal in the outer retry too, not only the inner same-FID loop
---------
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* mount: compare permissions against local ids without mapping them again
Access, open and create (without -defaultPermissions) ran FilerToLocal on
entries that maybeReadEntry and maybeLoadEntry already return in local ids.
Under a uid/gid map where a local id is also a mapped filer id, the owner
was mapped twice and the check refused the file's own owner.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* mount: cache a deferred create's placeholder in filer ids
The meta cache stores filer uid/gid and maps them to local ids on every
read, but createRegularFile cached the deferred or writeback placeholder
with the caller's local ids. The second mapping in the permission checks
used to cancel that out under a swapping id map; without it the owner of
a placeholder was refused. Cache a mapped clone and keep the handle's
entry in local ids.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* mount: test that a deferred create returns its entry in local ids
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* filer: stop following a peer that stays unreachable
* filer: re-track an answering peer only while it is still followed
A signature read that succeeds just before the master removes the peer
called initPeerWatermark after the removal, clearing its mark. The loop
then exits, so the watermark it re-tracked never advances or drops and
pins aggregated readers at the settled horizon.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* filer: forget a peer's unreachable clock when the master removes it
Only a successful connection cleared the clock, so every departed address
that once failed stayed in the map for the filer's lifetime, and a re-added
peer inherited the old outage. The removal now drops it, and the clock is
guarded by peerChansLock so the removed peer's exiting loop cannot start a
new one.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* filer: report the unreachable mark only when it is set
Once peerResumeRetention forgets the mark of a peer that stays
unreachable, markPeerWatermarkRemoved no longer marks it (its watermarks
are gone), but notePeerUnreachable still saw no mark and logged
"treating it as removed" on every retry. markPeerWatermarkRemoved now
reports whether it set a new mark.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* test: fix the unreachable-peer subscriber's start before the baseline write
The subscriber goroutine picked its start time itself, so a baseline write
that finished first fell before the subscription and the test failed after
20 seconds with correct delivery.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* mount: answer an interrupted read with EINTR, not EIO
The kernel's FUSE contract (Documentation/filesystems/fuse/fuse.rst,
"Interrupting filesystem operations") lets a filesystem either ignore an
INTERRUPT or honor it by replying EINTR to the original request. Read did
neither: the FUSE cancel channel cancels the read's context, the chunk
fetch fails with "context canceled", and every error became EIO.
Interrupts come from any signal the reader handles. Go's runtime sends
SIGURG for goroutine preemption, so Go programs reading under load hit
this on healthy data: on a 128-thread 64 KiB random-read test about 250
of 5 million reads failed with EIO, and none with GODEBUG=asyncpreemptoff=1.
Read now returns EINTR when its own context was cancelled, the way
dirReadStatus already does for directory reads and lookups do since
#11717. Go's os.File retries EINTR, so those readers see no error. A
cancelled read also skips the entry refresh retry, and neither read log
line reports an interrupt as a warning or error. Failures without an
interrupt stay EIO.
Tests: TestReadInterruptedReturnsEINTR (fails on the old code with EIO),
TestReadFailureWithoutInterruptReturnsEIO; weed/mount/... passes.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* mount: require the failed-read test to reach the filer
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* mount: skip the read retry once the request is interrupted
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
* mount: move an open file's read view to the chunks a merge wrote
When a flush finds that a file's chunks take more than twice its size,
maybeMergeChunks reads the file back and writes it again as new chunks.
The commit that publishes them makes the filer delete the chunks they
replaced. The flush put the new chunks into the handle's entry but left
its read view on the old ones, so every descriptor that kept the file
open went on reading the replaced chunks. Those read back until their
volume is vacuumed. After that a read first retries the fetch of the
missing needle for about 13 s; then refreshEntryForRead installs the
filer's entry and the read is retried. The refresh skips a handle with
writes it has not flushed, and such a read returns EIO instead.
Files that are kept open and rewritten in place, such as PostgreSQL
data files and SQLite databases, are the ones whose chunks bloat, so
they are the ones that hit this.
installRewrittenChunks puts the merge's chunks into the entry and the
read view together, under the entry lock. Chunks attached while the
merge ran were not part of it and stay. Flushes without a merge are
unchanged.
TestMergeMovesReadViewOffReplacedChunks covers the handle.
TestReadOpenHandleAfterChunkMerge runs it on a real mount, with and
without an unflushed write; for it, the fuse_integration framework
starts mini on a free master port and exposes the address.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* mount: drain uploads before a merge replaces the chunks
Uploads attach their chunks without the handle lock, first to the entry
and then to the read view, and writes that landed after doFlush drained
the pages may still be uploading when the flush takes its lock. An
upload that attached during the merge could keep a chunk older than
data the merge already holds, or put a chunk the merge replaced back
into the read view. Either outranks the merge's zero-timestamp chunk.
When the chunks are bloated enough to merge, the flush now drains the
pages again while its lock holds writes off, and takes the chunks again.
FlushAll returns only after every upload has attached to both the entry
and the read view, so nothing attaches during the merge.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* mount: leave a pre-merge upload failure to the next flush
The drain before a merge returned an upload failure through the
metadata flush. doFlush then retried it as a filer error and answered
EIO, where the page flush answers a full disk with ENOSPC. The drain is
there only to wait for the uploads: a failure stays with the pages, and
the next flush reports it as a write error, as it would without a merge.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* mount: keep chunks attached while a flush installs its compacted list
A write that lands after doFlush drained the pages can still be uploading
when flushMetadataToFiler takes its chunk snapshot. Its upload attaches the
chunk without the handle lock, and the plain flush then replaced the
entry's chunks with the list compacted from the snapshot, dropping it from
the handle and from every later commit. Keep chunks attached after the
snapshot, as the merge path already does.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* mount: fail a merge's flush on a failed drain
A failed upload found by the drain before a merge was only logged, and
the flush went on to publish a size covering bytes that no chunk stores.
It now fails the flush as a pageUploadError before anything is
published; doFlush maps it with writeErrorToFuseStatus, as it does its
own FlushData failure, and does not retry it as a filer error.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* mount: clone a flush's request entry under the entry lock
The request snapshot cloned the live entry after GetEntry had released
its read lock. Uploads attach chunks without the handle lock, so an
AppendChunks that landed during the clone raced it on the Chunks field.
Clone inside UpdateEntry, as content flush recovery does.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* mount: leave a merge to a later flush while uploads are pending
Before a merge the flush drained the pages while it held the handle and
publication locks. A writeback upload retries a transient storage error
until it succeeds, so that drain could hold both locks for as long as
storage stayed down, stalling reads, writes and stat on the handle.
The flush now checks for pending uploads and, if any, publishes the
compacted chunks and leaves the merge to a later flush. With none
pending the lock still holds writes off, so nothing attaches during the
merge, and a failed upload still fails the flush as a write error.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* mount: keep room for pending uploads when a flush merges onto a moved entry
A flush that leaves the chunk merge to a later flush publishes while
uploads are still pending, and writes that landed after the drain can be
uploading on any flush. Merging a peer's inline content or a shrink into
such a publication lets the upload attach its chunk next to the inline
bytes, where reads never reach it. Tell createContentEntry when uploads
are pending so the merge keeps room for them.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* mount: test chunk merges through the filer's chunk proxy, also under load
The tests serve the filer's chunk proxy, and like the filer the fake
deletes the chunks each commit replaces, so a read view still naming
them fails:
- TestFlushMergeThroughChunkProxy/Quiet: the flush merges, and reads
through the open handle return the file from the merged chunk alone.
- TestFlushMergeThroughChunkProxy/UploadInFlight: an older page is still
uploading while a newer page of the same bytes has attached. The flush
leaves the merge, reads return the newer page, and the next flush
merges.
- TestFlushMergeRacesWithWritesAndReads: four writers, three flushers
and a reader on one handle, with uploads delayed at random. The
writers pause now and then, so 14 to 18 flushes a run merge and the
others leave the merge to a later one. Every read must succeed within
2 s, and at the end the handle and the filer must both hold exactly
what was written.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* mount: let a held upload go when UploadInFlight fails
The subtest's proxy holds the older page's upload until the test
releases it, and only the passing path did. A failure in between left
the handler blocked, so the proxy's cleanup waited on it and the failure
surfaced as a suite timeout. Release it from a cleanup as well, once,
and bound the waits for the upload and the drains.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
* mount: ask the filer again when a deferred create is published mid-lookup
A program that writes a file under a temporary name and renames it while
the file is still open (safetensors save_file does, through Rust
tempfile's persist) looks the temporary name up before the rename. When
the parent directory's listing is not cached, that lookup goes to the
filer. If a FLUSH publishes the deferred create while the lookup is in
flight (a forked child closing its inherited descriptor sends one), the
filer can answer not-found from a read taken before the create
committed. By the time the answer arrives the flush has cleared the
handle's dirty flag, so lookupEntry finds nothing local to explain the
answer, logs "possible coherence bug" and returns ENOENT, and the rename
fails although the file exists.
Note before sending the request whether the inode had unpublished
metadata (a dirty handle or an async flush in flight). On not-found, if
it had and has since been published, ask the filer once more: the filer
acknowledged the publish before the second request is sent. A second
not-found stands, so a file another mount removed is not brought back.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* mount: judge a placeholder's not-found by one look at its base
Review of the previous commit found two problems. The retry check and the
not-found fallback read the handle separately, so a flush that finished
between the two reads still turned the answer into ENOENT. And the
pre-request read of dirtyMetadata, a plain bool that Write and the flush
set under the handle lock, raced with them on every lookup that reached
the filer.
Note instead whether the handle is a local placeholder: a deferred create
has no filer base (baseEntry) until a flush publishes it. A not-found for
a placeholder is judged by one atomic look at that base. With none, the
filer cannot have the file yet and the handle answers. With a base, the
filer acknowledged the publish, so ask once more. dirtyMetadata is not
read for this, and the not-found fallback keeps the existing check, with
the placeholder decision added.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* mount: answer a placeholder from its own handle; a second miss is final
Follow-ups from review of the previous commit:
- Pick the placeholder after the async-flush wait, from the path's
current mapping.
- A placeholder that still has no base answers from the observed handle
itself, instead of through the not-found fallback, which re-resolved
the path and could miss once the handle was released and the cache
entry evicted.
- A not-found from the second request is final. The flush installs the
base before it clears dirtyMetadata, and keeps it set when a write
landed meanwhile, so the fallback could return a file another mount had
removed.
- Test gates release on cleanup, so a failing test leaves no handler
blocked.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* mount: read a placeholder's entry the way GetAttr does
The previous commit answered a not-found for a placeholder from the
handle's live entry under entryLock. Neither of the entry's writers takes
that lock: Write and the metadata flush change size, times and chunks
under the handle lock, and async upload workers append chunks under the
LockedEntry lock, so the lookup could read an entry while it changed.
Answer with the copy the create cached when it is still this file's
(same inode), as the not-found fallback already prefers. Otherwise copy
the handle's entry under both writers' locks, in GetAttr's order (handle
lock shared, then the LockedEntry lock), and convert the copy. No caller
of lookupEntry holds a handle lock.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* mount: cache a deferred create's placeholder in filer ids
The meta cache stores filer uid/gid and maps them to local ids on every
read, but createRegularFile cached the placeholder with the caller's local
ids. Under a uid/gid map whose local id is also a mapped filer id, a lookup
or stat answered from the placeholder reported the wrong owner. Cache a
mapped clone and keep the handle's entry in local ids.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* mount: assert the placeholder is cached before checking its owner
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* mount: check the snapshot owner after evicting the cached placeholder
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
* helm: set publishNotReadyAddresses on the volume headless service
* fix(helm): separate volume DNS discovery from ingress traffic
* test(helm): cover volume DNS discovery and ingress routing
* fix(helm): rename the volume routing Service so volumes.client cannot collide
* docs(helm): name the ready-only Service for explicit volume backendRefs
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* fix(helm): fail when the volume routing Service name collides after truncation
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* docs(helm): point explicit volume backendRefs at the rendered client-volume name
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* volume: remove files a failed VolumeEcShardsCopy wrote
The shard files were copied before the .ecx, so a source holding a
0-byte index failed the request only after its shards had landed, and
nothing removed them: every retried move stranded another unmounted
shard on a new disk.
Copy the .ecx first, staged beside its target, so a bad index fails
before any shard lands; rename it into place only after the shards and
sidecars have copied, so a failed copy never truncates or replaces an
.ecx the destination already has. On any error, remove exactly the
files this call created; files already on the disk (an existing
holder's shards, index or journal) are left alone. Go and Rust volume
servers behave the same.
Claude-Session: https://claude.ai/code/session_011UpvafcfUuC2mLxNAXLQJi
* ec_balance: skip EC volumes whose every holder reports an empty .ecx
A shard copy refuses a 0-byte .ecx, so detection proposed the same
failing move of such a volume every cycle. Leave volumes with no
holder reporting a needle out of planning; their shards still count
toward disk occupancy.
Claude-Session: https://claude.ai/code/session_011UpvafcfUuC2mLxNAXLQJi
* volume: serialize VolumeEcShardsCopy per volume
Overlapping copies of one vid shared the staged .ecx, and a failed copy's
cleanup removed shards that a concurrent copy had already committed and
mounted. Hold a per-vid EC lifecycle lock across the whole copy and its
cleanup, named as in the enterprise build (Store.EcLifecycleLock /
ec_lifecycle_lock).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* volume: overwrite an existing .ecx in place when committing a copied index
A mounted EcVolume reads its index through an open handle, so renaming
the staged copy over it left the reader on the replaced inode: tombstones
the source had folded into its .ecx never reached the mounted holder.
An index already on the disk is now overwritten in place, as before the
staging change; a fresh destination still gets the rename. Go and Rust.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* volume: stage replacements of shards a VolumeEcShardsCopy finds on disk
The shard transfer opened an existing destination shard with O_TRUNC and
removed it on a stream or write error, so a failed retry deleted a shard
this disk already held. A shard already present is now copied to
<shard>.copying and renamed into place only after every other file has
copied; a mounted copy is reopened so it serves the new file. Fresh
shards still write in place and are removed on failure. Go and Rust.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* volume: stage the .vif and .ecsum a VolumeEcShardsCopy brings until commit
The sidecars landed as soon as they transferred, ahead of the deferred
.ecx commit, so a failure after them left a new .vif beside the index
the failed copy kept. They are now staged as .copying and renamed into
place with the shards and index once every transfer has succeeded; a
source without one still stages nothing. Go and Rust.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* volume: run a Rust VolumeEcShardsCopy to completion when its caller goes away
tonic drops a cancelled handler future at its current await, which
released the EC lifecycle lock while file operations the copy had started
(a cleanup remove_file on the blocking pool, a create or rename) could
still land; a retry taking the lock could then lose its files to them.
The copy now runs in a spawned task that owns the lock, as Go's handler
runs to completion.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* volume: fail the sidecar copy tests with an unreadable source .ecsum
A directory at the source path reads as "not found" on volume servers
that skip non-regular EC files, which turned the failure into a no-op.
A mode-000 file fails the transfer on every implementation; root and
Windows, which can read it, skip the test.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* volume: copy each shard a VolumeEcShardsCopy lists once
A shard listed twice and already held here was staged twice; the first
commit rename consumed the staged file, so the second failed the request
after the shard was replaced, and every retry failed the same way.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* volume: wait out an in-flight VolumeEcShardsCopy before mounting its shards
A shard the copy had already landed could be mounted while the copy was
still running; the copy's failure cleanup then removed the file but left
the shard registered and announced. MountEcShards takes the per-volume
lifecycle lock, as the enterprise build already does, and the Rust mount
handler takes the same lock.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* volume: fail a VolumeEcShardsCopy whose replaced mounted shard cannot be reopened
The copy reported success while the mounted shard kept serving the
replaced file. The error lets the caller retry the copy instead of
treating the shard as delivered.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* volume: check which transfer fails in the EC copy cleanup tests
An earlier failure, before any shard or sidecar landed, would leave the
destination unchanged and pass every assertion without exercising the
cleanup. The sidecar test also skips on Windows, where a mode-000 file
stays readable.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* volume: fail the sidecar transfer in the EC copy test by injection
A mode-000 file is still readable to root and on Windows, so the test
skipped there. The source now returns an error for the .ecsum request
only, and the test checks that it served .vif before that failure.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* volume: take the EC lifecycle lock for a blanket full teardown
A teardown acknowledged while a detached copy of the same volume was
still waiting on its source let that copy recreate the index and shards
afterwards. Go and Rust now wait for the copy, as the enterprise build
does.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* volume: keep the request id on a Rust EC shard copy
The copy now runs in a spawned task, outside the gRPC layer's request id
scope, so its CopyFile calls to the source went out with a new id. Carry
the caller's id into the task.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* volume: unmount a replaced EC shard whose reopen fails
The failed copy left the mount reading the replaced file's inode, so the
destination kept serving the old shard. Unmount it, which also reports
the removal to the master, so a retry copies and mounts it again.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* volume: make the Rust EC copy tests fail for the right reason
The sidecar test now serves a directory as the source .ecsum, which root
cannot read either, so it no longer returns early there. The cancel test
holds the store until the handler's join reports the cancellation, so a
copy that finished before the abort can no longer pass it.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* volume: stage every shard a VolumeEcShardsCopy transfers
A shard missing here was written in place with a cleanup guard armed
before its transfer started, so a failed copy removed a shard that
ReceiveFile wrote meanwhile. Every shard is now staged, and only one
that did not exist when it is moved into place is removed on failure.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* volume: take the EC lifecycle lock for VolumeEcShardsRebuild
A rebuild folds the deletion journal into .ecx. Run during a copy of the
same volume, the copy's index then replaced the folded tombstones. Go
and Rust take the lock the enterprise build already takes here.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* volume: check the staged .vif bytes in the EC sidecar failure test
A source without the optional .vif also returns success, so the served
extension list could not prove the transfer. The test now reads the
staged .vif when the .ecsum request fails and compares it with the
source's.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* volume: unmount a failed reopen's shard on the copy's disk only
UnmountEcShards walks every disk, so a reopen failure on the destination
also unmounted a healthy copy of the same shard on a sibling disk. The
copy now unloads the shard from its own disk and reports that disk's
removal.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* volume: take the EC lifecycle lock for an ordinary EC shard delete
Shards a copy has staged are invisible to the delete's last-shard check,
so deleting the only canonical shard removed the index the copy was
about to commit its shards against. Go and Rust deletes now wait for an
in-flight copy of the same volume.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* volume: leave a mounted shard's copied file in place on copy failure
A shard mounted on the copy's disk whose file was missing got the copied
file reopened into its mount, then that file was unlinked by a later
failure's cleanup while the mount kept serving it. Only an unmounted
shard's new file is removed now.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* volume: skip repeated shard ids with a seen set
Scanning the request prefix for each id made a request with many
repeats quadratic while the copy held the volume's lifecycle lock.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* volume: make the Go EC copy and lock tests observe their effects
The fake source counts requests per extension, so the duplicate-id test
proves .ec05 is fetched once, and returns fixture errors instead of
failing the test from its handler goroutine. The staged .vif reaches
the test through a channel. The mount and teardown lock tests seed real
files and check that nothing happened while the lock was held and that
the operation took effect after it was released.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* volume: poll the handler directly in the Rust EC lifecycle lock tests
A spawned handler that had not run when the 200 ms timer fired passed
the blocking check without ever reaching the lock. The tests now poll
the pinned handler future themselves, which guarantees it is waiting on
the lock, and cover the ordinary shard delete as well.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* volume: observe the lifecycle-lock waiter in EC copy lock tests
The mount, teardown, shard-delete and rebuild tests slept 200 ms while
holding the lock, so a handler scheduled only after the unlock passed
without ever taking it. Wait until the handler's goroutine is blocked on
the mutex instead; each test now fails within its timeout if the handler
skips the lock.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* volume: check the shard is staged before the failure-path write in a copy test
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* volume: recover a missing EC index under the volume's lifecycle lock
Missing-index recovery fetched the index and then ran a store-wide mount
of every orphan shard, without the per-volume lifecycle lock. It could
mount shards an in-flight VolumeEcShardsCopy of another volume had just
renamed in, and that copy's failure cleanup then unlinked a mounted shard.
Each recovered volume now holds its lifecycle lock across a re-check, the
fetch and a mount restricted to that volume.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* volume(rs): recover a missing EC index under the volume's lifecycle lock
Mirror of the Go change: recover_missing_ec_indexes takes the volume's
ec_lifecycle_lock, re-checks that the index is still missing, fetches it,
and mounts only that volume's shards through mount_recovered_ec_shards_only,
so an in-flight VolumeEcShardsCopy of another volume never has its
renamed-in shards mounted from under it.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* volume: mount a volume whose index a copy installed during recovery
Recovery re-checks under the lifecycle lock that the index is still
missing. When a copy installed it in the meantime, the re-check skipped
the mount too, and the volume's other shards stayed unmounted because
later scans no longer list it. Skip only the fetch and still mount the
volume's orphan shards under the lock, in Go and Rust.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* volume: fetch a recovered EC index into the disk the locked re-check found
The re-check under the lifecycle lock only kept a yes/no answer, so the
fetch still wrote to the destination from the first scan, which may be a
disk that has since gone away. Fetch into the record the re-check returned.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* volume: mount a copy-installed EC index even when no peer is known
Recovery skipped the locked step when the master was unreachable, the
lookup failed or no peer was listed. A copy that installed the index after
discovery then left the volume's other shards unmounted, and later scans
no longer list it. Always run the locked re-check, and give up for lack of
peers only when the index is still missing.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* volume: grow the stack buffer before scanning for a lock waiter
A full buffer can cut off the handler goroutine's stack, so the test would
report a handler that is waiting as one that never waited.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* volume(rs): fetch a recovered EC index into the disk the locked re-check found
Mirror of the Go change: fetch into the record the re-check under the
lifecycle lock returned instead of the destination from the first scan.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* volume(rs): mount a copy-installed EC index even when no peer is known
Mirror of the Go change: a failed lookup or an empty peer list no longer
skips the locked re-check, so a volume whose index a copy installed after
discovery still gets its orphan shards mounted.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* volume(rs): run missing-index recovery to completion when the mount RPC is cancelled
Dropping the mount handler mid-recovery released the volume's lifecycle
lock while the .ecj merge it had started on a blocking thread was still
landing, so a teardown could remove the volume before that late write.
Run recovery on its own task, as the copy handler does, so the lock is
held until it finishes; Go's handler already runs it to completion.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* volume: skip other volumes before stat in a single-volume EC scan
Recovery re-checks and mounts each volume with its own scans, so a
request with many orphans repeated whole-store discovery once per volume.
The index-owner and orphan-shard scans now take the same volume filter
and drop other volumes' files by name, before the per-file stat, and the
locked re-check scans only its own volume.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* volume(rs): skip other volumes before stat in a single-volume EC scan
Mirror of the Go change: the index-owner and orphan-shard scans take the
volume filter and drop other volumes' files by name before the per-file
stat, so recovery's per-volume re-check and mount no longer repeat
whole-store discovery under the store lock for every orphan.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* volume(rs): announce shards recovered after the mount RPC was cancelled
Recovery now outlives a cancelled mount RPC, but the heartbeat nudge
stayed in the handler, so shards it mounted after the caller left waited
for the next full EC heartbeat. Nudge it from the recovery task, as Go's
loads announce each recovered shard.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* volume: match the collection as well as the volume id in the recovery re-check
Discovery keys orphans by (collection, vid), but the locked re-check
matched the vid only. Two collections with orphans of the same vid could
both resolve to the first record, so the other's index was never fetched.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* volume(rs): match the collection as well as the volume id in the recovery re-check
Mirror of the Go change.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* s3api: answer NoSuchBucket from GetBucketRequestPayment for a missing bucket
The handler wrote the static BucketOwner answer without resolving the
bucket, so a bucket that does not exist looked like one that does.
Check the bucket first, as PutBucketRequestPayment and the other static
configuration getters already do.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3api: test GetBucketRequestPayment on existing and missing buckets
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* filer: release the stamps of parent directories a rename creates
* filer: test that a suppressed create keeps the moved entry's stamp in flight
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* operation: prepare upload content before assigning the volume
* operation: keep the existing upload function names for prepared payloads
UploadData and doUpload prepare once and hand the prepared payload to
retriedUploadData, so the retry loop, doUploadData, uploadChunk and
uploadChunkToHolders take a preparedUpload directly instead of growing
a prepared twin of each and branching on nil.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* operation: send ciphertext whenever the prepared payload carries a cipher key
doUploadData chose the cipher branch from the option, so a payload
prepared under a different option could be sent as plaintext metadata
or lose its key. Decide from the prepared payload itself.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* operation: build a chunk's preparation and upload options in one place
The chunk loop prepared with a hand-written option while uploadChunk and
uploadChunkToHolders built their own, so a setting that shapes the bytes
could reach one and not the other. chunkUploadOption now feeds all three.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* operation: test that a reassigned chunk reuses its prepared payload
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
The Go volume server fails the RPC when the remote read returns a
different length than req.Size; the Rust server cached whatever came
back as a full-size chunk.
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* wdclient: let master-pushed adds replace the gateway's lookup-fed locations
After a master leader change the S3 gateway clears its FilerClient cache and
refills it through filer lookups. The filer's own master client may still
answer from the previous leader's generation until the new leader has heard
from the volume servers, so a volume server lost together with the old leader
is cached again. The new leader never knew that server and never pushes its
removal, and the gateway keeps trying it until reads fail on it volume by
volume, one 30 s dial timeout at a time.
The gateway's master client relearns every volume from the new leader's
adds. On each add, cache the volume where the master client now knows it to
be, in place of what a lookup cached, and keep lookups already in flight from
caching their answer over it. A provider answer for a volume is now cached
whole or not at all.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* wdclient: take a lookup's fill epoch before its second cache check
A master add that landed between the second cache check and the read of
the fill epoch was invisible to both: the check had already missed, and
the epoch the lookup then read already included the confirmation, so a
filer answer that still named the lost server was merged back into the
confirmed entry. Taking the epoch first means such an add either shows
up in the check or stops the answer from being cached. The same order
covers a reset or clear in that gap.
TestConfirmDuringLookupIsNotUndone fires a confirmation from a test hook
at that point and fails with the old order.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* wdclient: gate the master lookup provider's cache writes on the map epoch
The provider wrote each answer straight into the vid map, so a lookup sent
before a master reset and answered after it landed in the new generation,
which a gateway's FilerClient now reads as the current master's view. The
answer now goes through fillLocations, which drops it once a reset passed.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* wdclient: keep the vidMapClient doc comment on its type
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
* s3: answer ClientDisconnected for a GET the client aborted before the first byte
Fixes#12706
* s3: tell a gone client on TLS and HTTP/2 too, and not on a connection the server closed
* filer: fail health check when absent from lock ring
If the master broadcasts a lock ring that does not include this filer
(e.g. the late-cleanup race in seaweedfs#11442), every lock acquisition
fails with "no lock server found" and locked writes, including S3 PUTs,
hang indefinitely while /healthz and /readyz still report OK.
Make the filer health handler return 503 once this filer has been
missing from its own lock ring for longer than a 30s grace period, so
Kubernetes liveness probes restart it and it re-registers with the
master. The grace period covers startup before the first ring update
and the master's ring-change batching delay.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* filer: hold the lock-ring health clock until the filer announces
AnnounceOnceCaughtUp can hold the master registration for up to a minute
while HTTP already serves, so the filer cannot be in the ring yet. Report
healthy until the announce gate opens and start the grace period after it.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* filer: keep a probe racing a ring rejoin from reading a reset clock
A probe that loses the CompareAndSwap to a concurrent healthy probe could
load the reset zero and report the filer absent since the epoch.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* filer: trim the lock-ring health comments
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* filer: restart the lock-ring health grace when a ring update brings the filer back
The clock only reset when a probe saw the filer in the ring, so a rejoin
and a second removal between two probes reused the first absence and
failed the next probe early. Reset it from the ring snapshot callback.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* filer: test the announce gate leaves the lock-ring health clock unset
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* filer: re-read lock-ring membership after starting the health clock
A probe whose first read raced a rejoin could start the clock after the
snapshot callback reset it, leaving a stale start for the next removal.
The ring is installed before the callback runs, so a second read sees the
rejoin whenever the reset came first.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* filer: treat a health clock changed during the probe as a rejoin
A rejoin and a second removal can both land between the probe's two
membership reads. The second read then shows the filer absent again, and
the probe judged the new absence by the expired start it had read before
the rejoin's reset. The snapshot callback resets the clock before the next
ring update is installed, so a clock that no longer matches the probe's
start means a rejoin happened; report healthy and let the next probe use
the fresh clock.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* filer: test ring updates that land during a lock-ring health probe
Move the absence clock into lockRingAbsence, which takes the membership
read as a function, so the test can install ring updates between the
probe's reads: a rejoin after the first read, and a rejoin plus removal
between the two reads.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* filer: give every lock-ring health reset a new clock value
A probe that saw the filer absent started the clock with a CAS from 0,
so a rejoin that reset the clock to 0 again before that CAS was
invisible, and the probe could install a start time from before the
rejoin. Each reset now stores a new non-positive value and the probe
swaps from the value it loaded before the membership read, so any
rejoin during the probe makes the CAS fail. This drops the second
membership read and never waits on the LockRing mutex.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* filer: test lock-ring rejoins before and after the probe's membership read
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* filer: keep the lock-ring version lookup out of the unhealthy health path
LockRing.Version takes the ring mutex, which SetSnapshot holds while
the snapshot callback transfers locks, so an expired-absence probe
could wait on those transfers and time out instead of answering 503.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* filer: test an expired-absence health probe while the ring mutex is held
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
* test(fuse): two mounts staging the same files without -dlm
Two jobs that extract the same archive into one shared directory, each
through its own mount and without -dlm, both create many of the same
files: a file that exists is skipped, otherwise it is created, written
and closed, then chmod'ed and given its mtime. Before #12593, #12594 and
#12596 the attribute update and the content flush sent the mount's whole
cached entry. Sent after the other mount had rewritten the file, the
update put back chunks that the rewrite had dropped and the filer had
deleted, so the entry pointed at deleted needles and the file failed to
read once its volume was vacuumed. Nothing told the writers.
TestStageSameFilesWithoutDLM runs that on the fuse_dlm cluster with the
mounts started without -dlm (startTestCluster takes a dlm flag;
startDLMTestCluster is unchanged) and checks every staged file: each
chunk is still on the volume server (HEAD on the fid, without
readDeleted) and the filer serves the bytes written. What the losing
writer is told depends on the version and is only logged.
With a 4.53 weed binary the test fails: 66, 71 and 70 of the 400 files
were broken in three runs. With master it passes in about 48 s.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* test(fuse): bound the staging test's filer and volume checks
The checks used the default HTTP client and a background context, so a
server that accepts a request and stops answering held the test until the
package timeout instead of failing the check.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* test(fuse): fail the staging test unless both mounts wrote the same files
A mount whose MkdirAll failed returned early, and a mount that started
late could skip every file, so the test passed with one writer. Start both
loops together, fail on setup errors, and require files both mounts wrote.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* test(fuse): bound the staging test's checks with one deadline
A per-call timeout still let 400 hung lookups and reads run for hours.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
* s3: keep serving for -preStopSeconds after SIGTERM
On SIGTERM the gateway closed its listeners at once, so requests a load
balancer or Kubernetes Service still routed to it failed until the endpoint
was removed. With -preStopSeconds set, the gateway makes /status, /healthz
and /readyz return 503, keeps serving for that many seconds, then runs the
existing graceful shutdown. 0 (the default) keeps the old behavior.
The flag is registered for weed s3 and as -s3.preStopSeconds for weed
filer, weed server and weed mini. A cancelled weed mini cluster context
ends the wait early. The Helm chart passes s3.preStopSeconds and
filer.s3.preStopSeconds through.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3: assert the preStopSeconds chart render under set -e, drop timing from the probe test
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3: wait before weed mini cancels its clients; size grace periods in the chart
weed mini's interrupt hook runs before the S3 one and cancels the S3
gateway's shutdownCtx, so the pre-stop wait never happened there. The wait
is now also registered with beforeMiniClientsShutdown, which runs before
that cancel, and a sync.OnceFunc keeps the interrupt hook from waiting a
second time. Outside weed mini the registration is a no-op.
Helm: the s3 deployment fails to render when terminationGracePeriodSeconds
is below preStopSeconds + 15. The filer statefulset and the all-in-one
deployment, whose grace period is fixed at 60s, add preStopSeconds to it.
allInOne.s3.preStopSeconds is passed through, inheriting s3.preStopSeconds.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3: keep 64-bit counters aligned on 386; let allInOne.s3.preStopSeconds=0 disable the wait
draining sat ahead of inFlightDataSize and inFlightUploads in S3ApiServer
and shifted them off 8-byte alignment on 32-bit builds, so the atomic
upload-limit counters panicked there. It now follows them.
Helm: allInOne.s3.preStopSeconds inherits s3.preStopSeconds only when null;
default treated an explicit 0 as unset. The s3 values and the grace period
error now say a lifecycle preStop hook's time comes on top.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* grace: add OnInterruptFirst for hooks that must run before any shutdown
Interrupt hooks run in reverse registration order, so a hook cannot run
ahead of a service that registers later, such as a volume server that is
still loading its volumes.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3: run the -preStopSeconds wait before every other interrupt hook
Under weed server the volume server registers its hook once its volumes
are loaded, often after the S3 gateway. Its hook then ran first and shut
the volume server down, so the gateway spent the whole wait serving
requests that could not read or write objects.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
* s3: escape raw semicolons before the router matches a route
registerRouter applied EscapeSemicolonsInQuery with router.Use, but mux
runs router middleware only after it has picked the route. Queries()
matchers split pairs on ';', so GET /bucket?uploads;x=1 reached the
multipart listing handler while the handler and the IAM resolver read
uploads%3Bx=1 and saw no uploads subresource. Wrap the S3 router with the
escape so routing and authorization read the same query.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3api: report a listing without prefix as s3:prefix "" in policy conditions
The advanced IAM path already sets s3:prefix to "" when a bucket listing
names no prefix, as AWS does, but bucket policies and attached policies
read conditions from ExtractConditionValuesFromRequest, which leaves the
key out. The AWS home-directory policy that grants s3:prefix ["", "home/"]
then denied the root listing, and a Deny keyed on the empty prefix never
applied.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3: test the served handler composition for the semicolon escape
Move the escape and the Vectors split into s3Handler so a command test can
send ';' queries through the handler the server actually serves.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3: keep the isVectorsRequest doc comment on its function
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* fix(s3api): complete bucket-listing IAM condition handling
Populate s3:delimiter and s3:max-keys for bucket. Add regression tests for listing conditions and variants.
Assisted-by: gpt-6.1-sol, deepseek-v4.1-flash
* fix(s3api): ignore max-keys for multipart listing policies
Assisted-by: GPT-6.1 Terra
* s3api: drop max-keys from multipart-upload listing policy conditions
Bucket policies and embedded IAM policies read condition values through
ExtractConditionValuesFromRequest, which still exposed max-keys on ?uploads
listings. Those page by max-uploads, so an unused max-keys parameter could
satisfy an s3:max-keys condition. Match the advanced IAM path and leave the
key absent there.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* topology: never report negative free volume slots
Free volume slots are computed as max + remote - volume - ecSlots. When phantom or over-registered volumes push the count past the configured max, the result is negative and rolls up into the cluster total, causing volume grow and assign to fail with a negative 'volumes left' count even while healthy servers have capacity. Clamp every free-volume count at zero so an over-committed server contributes no slots instead of subtracting from the cluster.
* topology: drop the issue reference from the free-slot test comment
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* topology: test free slots next to an over-committed server
A server with more volumes than slots must not hide the free slots of
the other servers in its rack, data center or topology.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* topology: sum free slots per data node in racks, data centers and topology
Clamping the rolled-up counters only clamped their sum, so one server with
more volumes than slots still hid the free slots of every other server in
its rack, data center and topology. Interior nodes now add up their
children's free slots, so each data node counts as zero at worst. A data
node in maintenance mode no longer adds slots to its parents either.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* topology: test that disk info keeps an over-committed deficit
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* topology: keep the signed free volume count in disk info
EC encode retries and the balance planners add refunds and planned moves
to the published FreeVolumeCount. Clamping it at zero dropped an
over-committed disk's deficit, so a refund could make a full disk look
free. Volume growth already reads the clamped AvailableSpaceFor.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* topology: test placement next to an over-committed server
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* topology: accept either data center order in the placement spread check
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* seaweed-volume: expect a fresh redb file to start at 72 KiB
The assertion fails on redb 4.2.0, whose empty database is about 1 MiB.
* seaweed-volume: bulk-load redb rebuilds with the stable cursor
redb 4.6.0 has no experimental_cursor feature. Full rebuilds insert
sorted needle ids through insert_before, and a new .rdb starts at 72 KiB.
* seaweed-volume: test which redb files are still at the old floor
The gate is missing, so the test does not compile yet.
* seaweed-volume: decide when a reused redb is still at the old floor
Compact only a file above 72 KiB and at most 1 MiB plus 8 KiB, and only
when floor_compact has not been recorded.
* seaweed-volume: test one-shot compact of an old-floor redb
Reuse, shrink, and storage-error recovery are specified. The helpers
they call are not implemented, so the tests do not compile yet.
* seaweed-volume: compact a reused redb that is still at the old floor
One compact runs when the file is above 72 KiB and at most 1 MiB plus
8 KiB. A storage error drops the handle and opens the same path again
so the volume does not keep a PreviousIo latch.
* seaweed-volume: publish a redb replay before compacting the old floor
compact() is what would make a Durability::None tail durable, and a
storage error then drops that tail on reopen. An in-band file is
committed first. The floor_compact flag is read only inside the band,
so a meta error cannot rebuild a larger index.
* seaweed-volume: drop the redb experimental cursor CI step
redb 4.6 has no redb-experimental-cursor feature. The needle-map tests
that used it now run in the normal cargo test step.
* seaweed-volume: drop the experimental-api-5 notes on redb reads
redb's experimental-api-5 can no longer be enabled, so these reads always
use the inherent ReadOnlyTable::get.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* seaweed-volume: drop the always-false flag from the old-floor band check
Both callers pass false and read floor_compact on their own, so the band
check only needs the file length.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* master: unregister a volume server whose heartbeats stop
A volume server sends a heartbeat every 5 s. When its host hangs or its
network drops packets, its SendHeartbeat stream stays open, and the
master unregisters it only once gRPC keepalive closes the connection,
60 s plus 20 s after the last frame. Until then the master keeps
assigning writes to volumes with a copy on it, and those writes hang.
SendHeartbeat now ends the stream when no heartbeat arrives for three
pulses (15 s), which unregisters the server through the existing
deferred path and tells clients its locations are gone. The server
reconnects and registers again once it can reach the master.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* master: test that clients learn a silent server's volumes are gone
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* master: fail the silent-stream receive test instead of hanging
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
* pb: fall back to TCP when the local gRPC socket cannot be dialed
A temp cleaner can delete a local gRPC socket file while its server keeps
running. Every later dial to that server then failed with "no such file or
directory", so the master could no longer reach its own volume server and
volume growth stopped. Dial the server's TCP address when the socket dial
fails, as the S3 gateway already does for the filer's HTTP socket.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* pb: recreate a local gRPC socket whose file was removed
Once its socket file was deleted, a server kept accepting on an
unreachable listener until restart, so same-host RPCs stayed on the TCP
fallback. Check the path every minute and, when the file is gone, close
the old listener and bind the socket again.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* pb: retry a failed local socket rebind on the next check
A failed rebind ended the watcher, so the socket stayed missing after the
cause cleared. Bind the replacement before closing the old listener: a
failed bind keeps the old listener serving and is retried on the next
check, and the old Serve still reports server shutdown. The old listener
no longer unlinks the path on close, since the replacement now owns it.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* pb: set the local socket unlink policy before serving the listener
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* server: add -grpc.localSocketDir for the local gRPC sockets
`weed server` and `weed mini` hard-coded each local gRPC Unix socket
under /tmp. systemd-tmpfiles ages /tmp after 10 days and macOS after 3,
so a long-running server loses its socket, and RegisterLocalGrpcSocket
keeps routing local dials to the missing path: the master can no longer
reach its volume server.
Add -grpc.localSocketDir to both commands, defaulting to /tmp so an
existing setup is unchanged, and build every socket path from it with
filepath.Join. A folder that does not exist or is not a directory is
refused at startup rather than at the first local RPC. A folder whose
socket paths exceed the platform's sockaddr_un limit is refused too,
because every bind then fails with EINVAL while the server keeps running
and each local RPC dials a socket that does not exist.
Refs #12692
* server: refuse a -grpc.localSocketDir the server cannot bind in
An existing folder the server's user cannot write to passed the startup
check, and then every socket bind failed: ListenOnLocalSocket only logs
the error while GrpcDial keeps routing local dials to a socket that was
never created. After the path-length check, create and remove a probe
Unix socket in the folder, and refuse startup with the bind error if
that fails.
* server: give the -grpc.localSocketDir probe a random name
A probe named only by the process id could be precreated by another
user in a shared /tmp, or left by a crashed process whose pid was
reused, and the server cannot remove such a file, so startup failed
although the real socket paths were free. Name the probe with 8 random
bytes, retry a name that is taken, and rely on the listener's close to
unlink it.
* server: keep the -grpc.localSocketDir probe name within the service socket lengths
The probe was 5 bytes longer than the master and volume socket names, so a
folder whose service sockets fit the sockaddr_un limit could still be refused.
A 23-byte probe name is shorter than any service socket name, and a process
with no local sockets skips the check.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* server: trim the -grpc.localSocketDir comments and drop issue references
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Brent Pease <179639673+brentDSL@users.noreply.github.com>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* webhook: share the default target settings
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* webhook: managed webhook targets with a drainable queue
Targets kept as JSON (name, endpoint, token, filters, retry and worker
settings) build their own queue; Close waits for queued and in-flight
deliveries before stopping it.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* filer: run webhook targets stored in /etc/seaweedfs when notification.webhook.managed is set
The targets in /etc/seaweedfs/notification_webhooks.json start with the
filer and are rebuilt from the store whenever the file changes on any
filer, alongside notification.toml. The previous queue drains before it
stops. Events for the file itself, which holds bearer tokens, are not
sent to notification queues.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* admin: manage filer webhook targets
A Webhooks page under Management lists, adds, edits and deletes the
targets kept in /etc/seaweedfs/notification_webhooks.json. Bearer
tokens are write-only: the API reports whether one is set, and an empty
token on edit keeps the stored one.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* filer: keep the webhook queue when the stored targets did not change
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* filer: keep a replaced webhook queue open until its senders hand over
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* admin: read chunk-backed webhook targets before editing them
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* shell: keep the stored webhook targets out of fs.meta.notify
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* webhook: reject negative managed target settings
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* webhook: no signal handler on queues that reloads replace
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* filer: swap the webhook queue without waiting behind blocked senders
The read lock held across SendMessage made a reload wait for any sender
blocked on a saturated target, and the waiting writer stalled every
other filer write with it, which is when an operator disables that
target. Close now refuses new sends and waits for the senders already
inside SendMessage, and a send that hits the closing queue retries on
its replacement.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* webhook: release blocked test handlers before the server shuts down
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* filer: always run the webhook targets managed in the admin UI
Drop the notification.webhook.managed switch: every filer loads and live
reloads the stored targets. Without stored targets no queue is created.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* scaffold: place the admin-managed webhook note above enabled
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* scaffold: separate the admin-managed webhook note
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* scaffold: note that admin-managed webhooks follow the notification queue
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* filer: retry loading webhook targets after a failed read
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* filer: retire the admin-managed webhook queue on shutdown
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* ci: run the Python 3.9 jobs on 3.10
ubuntu-latest now resolves to Ubuntu 26.04, which has no Python 3.9 build
for actions/setup-python, so the Ceph S3 tests and the PyArrow Parquet
3.9 job fail during setup. 3.10 is the oldest version built for 26.04.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* test: list the Python versions the parquet CI runs
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* admin: apply the saved theme inline at the top of the page head
The theme script was an external file, so a reload had to fetch or
revalidate it before the root got its theme, and the page could paint
with the light default in the meantime.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* admin: set color-scheme on the root together with the theme
Until the stylesheets load, the browser paints the page canvas from the
root's color-scheme, which only the stylesheets set, so a dark theme
showed a white canvas for the duration of every reload.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* ec: mount shards against the index .vif when the local one has no EC config
A worker encode can send shards to a disk that still holds the source
volume's .vif while the encode's .vif lands with the index on a sibling
disk. The shard disk mounted through the source's config-free .vif, so
it reported encode time 0 and fell back to the sidecar layout, and reads
carrying the encode time were rejected as a different encode run.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* volume(rs): mount shards against the index .vif when the local one has no EC config
Mirrors the Go mount: a data-dir .vif that records no EC config yields
to the idx dir's when that one records it, and load_vif_info_across_dirs
reports the directory the .vif actually came from.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* storage: do not mirror EC sidecars onto a disk whose .vif has no EC config
Startup mirroring copied .ecx/.ecj next to a source volume's leftover
config-free .vif and kept that .vif, so after a restart the shard disk
mounted self-contained with encode time 0. Leave such a disk on the
cross-disk mount, which reads the owner's .vif.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* volume(rs): do not mirror EC sidecars onto a disk whose .vif has no EC config
Mirrors the Go startup mirror: a shard disk holding a source volume's
config-free .vif stays on the cross-disk mount, which reads the owner's
.vif, instead of mounting self-contained with encode time 0.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* ec: take the encode .vif from a sibling data dir when the index dir is shared
With a separate -dir.idx the .ecx is shared by every disk, but the encode's
.vif lands in the data dir of the disk that received it. A disk still holding
a source volume's config-free .vif mounted through it at generation 0.
NewEcVolume now accepts the server's data dirs and takes a sibling's EC .vif
when the index is not local and every sibling EC .vif names the same run.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* storage: give EC mounts the data dir of every disk on the server
NewStore records every data dir on each DiskLocation before loading, so the
startup scan, cross-disk reconcile and MountEcShards all pass them to
NewEcVolume.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* volume(rs): take the encode .vif from a sibling data dir when the index dir is shared
Mirrors the Go mount: EcVolume::new_with_vif_dirs takes a sibling's EC .vif
when this disk's own records no EC config, the index is not local, and every
sibling EC .vif names the same run.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* volume(rs): give EC mounts the data dir of every disk on the server
main sets Store.store_dirs before the first add_location, so a disk loaded
before its siblings still sees them; each DiskLocation passes them to
EcVolume::new_with_vif_dirs.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* storage: size startup EC validation by the sibling encode .vif
With a shared -dir.idx, a disk that still holds the source .dat and its
config-free .vif read the default data-shard count, judged custom-ratio
shards an interrupted encode, and removed them along with the shared .ecx
before the mount could take the sibling's .vif. The startup lookup now
falls back to the same sibling .vif the mount uses.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* volume(rs): size startup EC validation by the sibling encode .vif
Mirrors the Go change: validate_ec_volume falls back to the sibling data
dir's encode .vif before judging shards beside a source .dat.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* storage: keep EC shards when sibling .vif files disagree on the encode
SiblingEcConfigVif reports when a sibling .vif cannot be read or the
siblings name different encode runs. The startup data-shard lookup then
returns 0 instead of the default ratio, and validateEcVolume keeps the
files rather than sizing custom-ratio shards by a ratio they do not use.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* volume(rs): keep EC shards when sibling .vif files disagree on the encode
Mirrors the Go change: sibling_ec_config_vif returns Err for an unreadable
sibling .vif or disagreeing runs, ec_data_shards_from_vif returns 0 then,
and validate_ec_volume keeps the files.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* storage: release the opened EC shard when its volume cannot be built
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* volume: keep reads over the download limit local when readMode is local
With -readMode=local the server never sends a read for a volume it does
not hold elsewhere, but a read over -concurrentDownloadLimitMB on a
replicated volume was still redirected to another replica. Skip the
replica fallback in local mode, so the read waits for a download slot
and answers 429 when the wait times out.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* volume(rs): keep reads over the download limit local when readMode is local
Mirror the Go volume server: with readMode local, a read over the
download limit waits for a slot instead of being redirected to a
replica. proxy_or_redirect_to_target no longer needs the local-mode
redirect switch.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* volume: check download capacity under the wait lock
waitForDownloadSlot read the in-flight size outside the cond lock and
the read handler broadcast without it, so a download finishing between
the check and the wait was missed and the read waited out the timeout
for a 429. Check capacity with util.WaitUntil under the lock and
broadcast under it.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* volume(rs): register for the download notification before checking capacity
The wait loop created its Notified future only after the capacity
check, so a body dropped in between had its notify_waiters() lost and
the read waited out the deadline for a 429. Create the future first,
then check, as Go now checks under the cond lock.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* admin: fix cluster submenu toggle
* admin: update embedded JavaScript
* admin: drop the object store submenu handlers
The sidebar no longer renders an object store submenu, so its expand,
link and toggle handlers match nothing; the toggle handler repeated the
manual collapse that fought Bootstrap on the cluster submenu.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: ivan <i.stepanov@navigine.com>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
sudo switches its real uid to root while it starts, so kill -0 from the
invoking user fails with EPERM even though the mount process is alive,
and the wait loop reported "weed mount exited early" a few milliseconds
after starting it.
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* filer: serve caught-up aggregated subscribers from memory, not a held disk pass
An aggregated SubscribeMetadata stream that does not take metadata chunks
(filer.sync, filer.backup, filer.meta.backup, filer.meta.tail, the
filer.remote.* syncs, the S3 gateway's subscriptions, and mounts that
predate chunk support) re-ran the persisted-log pass whenever its cursor
moved: after every memory hold at the delivery watermark and on every
2 s re-probe. The pass holds at the peers' flush low-watermark. Peers
flush on their own one-minute schedules, so on a cluster that keeps
writing some peer has always flushed past a caught-up cursor while the
slowest has not: the pass held at its first entry, the loop came back
to it, and memory was never read. The stream got each change only once
the slowest peer's flush covered it, in bursts up to a flush interval
late, and every pass listed and re-read the newest log files.
When the aggregated ring was already filling at the cursor and evicted
nothing after it, memory holds everything the persisted logs could add,
since every peer's log reaches the ring through the aggregator. Skip the
pass and read memory, held at the delivery watermark as before. The
skipped pass proves nothing past the cursor, and the next-day scan for
an empty pass does not apply. An empty ring, a ring that starts past the
cursor, and a cursor memory just refused still take the pass. Chunk
subscribers already skip it for a caught-up cursor: their ref listing
stops below the flush hold.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* filer: keep a new peer's pre-start history on the disk pass
The ring only receives a peer's events from the start of its subscription.
A peer followed for the first time after this filer started (a filer added
to the cluster, a restarted pod with a new address, or the late-arriving
peers bootstrap) starts at its notice time, so what it logged before then
is only in the persisted logs. A cursor below that start passed the ring
check and skipped the disk pass, losing those events for subscribers that
do not take metadata chunks.
Record the newest start of a peer subscription that had no resume point and
send cursors below it through the disk pass as before. Peers that rejoin with
a resume point (a restart under the same address) do not move it.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
* helm: pin the 4.53 tag in the vector metrics rejection test
The 4.54 release made 4.54 the chart's default image, and that image
serves weed-vector metrics, so rendering vector.metricsPort with the
default tag no longer fails. Pin image.tag=4.53 for the rejection case
and check that the default image renders the metrics port.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* helm: describe vector metrics support from 4.54 on in the chart README
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* helm: say the vector metrics render guard skips image overrides
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* helm: drop the 4.53 vector metrics render guard
The chart now defaults to 4.54, whose weed-vector serves --metrics-port, so
the check keyed on one old tag only blocked valid installs. The test asserts
the default image renders the metrics flag and HTTP probes.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* helm: describe vector metrics without the render guard in the chart README
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* log_buffer: let a subscriber skip wake-ups for unwanted partition keys
Every append woke every registered subscriber. SetSubscriberWakeFilter lets
a reader sleep through appends whose partition key it would filter out; it
still reads them on its next poll. Flushes and keyless appends wake everyone.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* filer_pb: decide subscription matches from an event's directories
MetadataDirectoryMayMatchSubscription rules out events whose directory no
child path of can match the subscription, and ScanMetadataEventDirectories
reads an event's directory and new parent path without allocating.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* filer: wake subscribers of a move's destination directory
Appends are keyed by the source directory, so a filtered subscriber of the
destination would otherwise only see the move on its next poll.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* filer: wake metadata subscribers only for directories they can match
Every metadata event woke and was scanned by every subscriber, so filer CPU
grew with the number of subscribers watching other paths. Readers now
register a directory wake filter, and the per-entry prefilter rules entries
out from their directories before the skeleton scan.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* filer: test that moves into a subscribed prefix pass the prefilter
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* docs: explain the 3-filer recommendation in the production setup notes
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* docs: note that a filer load balancer must forward the gRPC port
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* s3api: create a multipart upload's directory on the object's owner filer
CreateMultipartUpload wrote .uploads/<id> to the gateway's own filer while
the upload's completion commits on the object's write owner. With
independent filer stores, a part or completion reaching another gateway
before peer metadata replication caught up could not see the upload.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3api: look up a multipart upload on the object's owner filer
UploadPart, UploadPartCopy, ListParts, CompleteMultipartUpload,
AbortMultipartUpload and the bucket policy SSE lookup read .uploads/<id>
from the gateway's own filer, so a request on another gateway shortly
after CreateMultipartUpload answered NoSuchUpload until replication
caught up. Read it preferring the object's write owner, like object reads.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3api: write multipart parts to the upload's owner filer
UploadPart and UploadPartCopy stored part entries on the gateway's own
filer, so ListParts and CompleteMultipartUpload on another gateway could
miss a part that replication had not delivered yet. Store them next to the
upload directory on the object's write owner.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3api: list and clean up a multipart upload on its owner filer
ListParts, CompleteMultipartUpload and AbortMultipartUpload listed and
removed part entries on the gateway's own filer. Do it where the upload
and its parts now live, so completion sees every part and cleanup does
not miss entries replication has not delivered yet.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3api: test a multipart upload against a lagging gateway filer
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3api: check the upload's owner before freeing a failed part's chunks
A part write that the owner stored but answered with an error went to
recovery with no object, so the owner was never among the filers checked.
When the owner is outside the gateway's filer list, the others' not-found
answers freed chunks the stored part still references.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3api: list an upload's parts on the prior owner after a ring change
The upload record lookup already falls back to the key's prior owner while
a ring change is in its cooling-off window, but the part listing read only
the new owner. ListParts then reported no parts for an open upload. Fall
back the same way when the new owner has nothing yet.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3api: find the upload record on the prior owner past a bare directory
A part routed to a new owner that has not received the upload yet makes
that filer create .uploads/<id> without the upload's key. The post-write
check then read it as an aborted upload and removed the part. While the
ring change is in its cooling-off window, a record on the prior owner
now wins over such a directory.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3api: merge the prior owner's parts into an upload listing
During a ring change's cooling-off window the new owner may have received
only some of an upload's parts. Listing returned the new owner's parts
alone once it had any, so ListParts dropped the rest and a completion
naming them failed with InvalidPart. Both owners' pages are now merged,
the owner's entry winning a shared name, and cut to the page limit.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3api: fail an upload listing the prior owner could not answer
When the new owner has none of an upload's parts yet and the prior
owner's listing fails, the parts may all be on the prior owner. That
listing read as empty, so CompleteMultipartUpload answered NoSuchUpload
for an upload that still exists; it now returns the error.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3api: fail an upload lookup the prior owner could not answer
During a ring change's cooling-off window an upload the new owner has
not received yet, or holds only as a bare directory, read as NoSuchUpload
when the prior owner's lookup failed. That failure is now returned, so
the client sees a retryable error instead of an upload that is gone.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* admin: support [maintenance.s3_lifecycle] in admin.toml
The admin.toml overlay covers vacuum, volume_balance and erasure_coding but
not s3_lifecycle, so the bucket lifecycle job's settings can only be changed
from the web UI and are lost whenever the admin data directory does not
persist. That is the worst combination for this job type: its
meta_log_retention_days knob decides whether bucket rules ever apply to
objects written before the job was enabled, and silently reverting it to the
default makes the rules no-op without any error.
Wire the four admin-side knobs into the existing overlay:
meta_log_retention_days retention window the meta-log replay can reach
walker_interval_minutes minimum gap between full bucket walks
cluster_deletes_per_second cluster-wide delete budget
cluster_deletes_burst token-bucket burst for that budget
All four are admin config values, not worker values: the engine partition is
computed on the admin side and the delete budget is divided across
execute-capable workers at dispatch time. enabled/retry_limit/
retry_backoff_seconds come for free from the shared section handling.
Tested with TestApplyPluginConfigFromTomlS3Lifecycle, which also pins the
retention=0 case: 0 means unbounded and must survive as an explicit 0 rather
than being treated as unset.
* admin: fix meta_log_retention_days guidance in admin.toml scaffold
Both reviewers caught a real error. `0` is not "unbounded": runShard does
retentionWindow := cfg.RetentionWindow
if retentionWindow <= 0 {
retentionWindow = maxTTL
}
so zero falls back to the longest rule TTL, which keeps every replay rule in
replay and promotes nothing to the walk set. Recommending 0 to force full
scans was wrong, and so was the claim that pre-existing objects are only
reachable through the walk set -- the cold-start recovery walk covers them
regardless of the retention value.
Describe zero as "no retention info supplied" and tell operators to set the
window the filer meta-log actually retains. The code change is unaffected:
it only wires the four keys into the existing overlay.
* admin: tie s3_lifecycle walk-set scans to walker_interval_minutes in admin.toml
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* admin: show the s3_lifecycle retry defaults in admin.toml
The s3_lifecycle descriptor sets no retry limit or backoff, so a failed
run is not retried and the scheduler's 5 s backoff applies; the scaffold
listed the vacuum values instead.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* admin: shorten the s3_lifecycle toml section comment
The retention and walker knobs are read by the worker from the admin
values, not computed on the admin side.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* admin: cover cluster_deletes_burst in the s3_lifecycle toml test
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* admin: check the s3_lifecycle retention zero is a stored int64
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* s3 test: evaluate mutation conditions and serve DeleteEntry in the fake filer
The fake ObjectTransaction now checks each mutation's own condition before
applying it, as the filer does, and reports the failed mutation. ETag clauses
read the stored ETag attribute first, like the filer. DeleteEntry removes the
entry so unrouted delete paths can run against the fake.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3 lifecycle: write the suspended expiration marker before deleting the null version
A suspended-versioning expiration deleted the null version first and then
wrote the delete marker. When the pointer read or the marker write failed in
between, the retry found no null version, resolved as NOT_FOUND or
VERSION_NOT_LATEST, and never wrote the marker, so an older version became
the latest.
The marker now goes first. The latest pointer outranks the null object, so
the key reads as deleted once the marker lands, and a null delete that fails
after it leaves a noncurrent null version rather than exposing an older one.
The null delete is pinned to the entry read before the marker: on the owner
it carries IF_ENTRY_EQUAL on that read, and off the ring the entry is re-read
and compared by identity first. A null object written after the read is a
newer write and is no longer deleted. When the null version is the one that
passed the expiration checks, a missing or changed entry at that read
resolves the request as STALE_IDENTITY before anything is written.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3 lifecycle: retry a failed routed null delete instead of deleting on the default filer
A routed null delete that failed fell through to the unrouted path, which
reads and deletes on the default filer. That filer may not have the owner's
newer null object yet, so the fallback could delete it without the owner-side
IF_ENTRY_EQUAL guard. The failure now returns RETRY_LATER; the marker is
already written, so the key keeps reading as deleted.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3 lifecycle: recheck and delete an unrouted null version under the object write lock
Without a ring owner the identity recheck and the null delete were separate
calls, so a suspended PUT landing between them was deleted. Off the ring
writers take the object write lock, so the recheck and the delete now run
under it.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3 test: strip quotes from the stored ETag in the fake filer
The filer normalizes both sides of an ETag comparison; the fake now strips
quotes from the stored ETag attribute as well as from the candidate.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3 lifecycle: retry the null delete when the object lock lookup fails on a routed key
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3 lifecycle: check the suspended marker against the validated version
A suspended expiration that validated a named version wrote its marker
without comparing the pointer, so a null PUT landing after the checks was
hidden, and deleted too when it landed before the null read. The marker now
compares the latest the way the enabled path does and resolves the request
as VERSION_NOT_LATEST.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3 test: count the marker's object write lock in the unrouted null delete
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3 lifecycle: recheck the attempt deadline after reading the null version
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* s3: accept the governance bypass header case-insensitively
botocore and the AWS CLI send x-amz-bypass-governance-retention: True,
so the exact "true" match ignored the bypass and governance-locked
deletes and retention changes failed with AccessDenied.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3: test governance bypass with SDK-style header values
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* s3api: normalize a nil policy value to no values
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3api: evaluate each policy condition against its own values
The process-wide normalized value cache keyed entries by fmt.Sprint of the
value, so ["a", "b"] and ["a b"] (or [] and [""]) shared one entry and a
condition in one policy could be compared against another policy's values.
Normalizing is cheaper than building that key, so drop the cache.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3api: test that look-alike condition values stay distinct
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* s3 lifecycle: bound each LifecycleDelete attempt with its own deadline
Replay and walker deletes used the job context with no request
deadline, so one stalled unary RPC could hold a shard for the whole
job. Each attempt now runs under DeleteTimeout (default 1m), set after
the limiter wait. A timeout while the job is still live is retried like
a transport error and then halts the shard or walk without advancing;
parent cancellation still stops both the attempt and the backoff.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3 lifecycle: admin setting for the delete RPC timeout
Adds delete_timeout_seconds to the S3 Lifecycle admin form and passes
it to both the replay drain and the walker. 0 keeps the 60s default.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3 lifecycle: describe replay and walker timeouts separately in the help text
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3 lifecycle: fail the pass when a replay delete exhausts its attempts
A transport error or attempt timeout that outlived the retries halted
the shard with a nil error, so the job reported success. processMatches
now returns the error (the cursor still holds at the event), and
runShard treats a drain error as a clean stop only when its own context
ended, since an attempt timeout also wraps DeadlineExceeded.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3 lifecycle: test that a replayed LifecycleDelete resolves after a lost reply
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3 lifecycle: skip the expiration write once the attempt deadline passed
A worker retries a LifecycleDelete whose attempt timed out, while the
first attempt can still be running on the S3 server. An attempt that
stalled before its marker write would then stack a second delete marker
after the retry wrote one. The gRPC deadline reaches the handler context,
so check it right before the expiration writes.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3 lifecycle: recheck the attempt deadline after the suspended null delete
The null-version delete can block on the owner filer past the attempt
deadline; the worker may have retried and written a marker by then.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3 lifecycle: finish the suspended marker once the null version is deleted
Skipping the marker after a successful null delete left an older version
as the latest, and the retry then resolved as a no-op without hiding it.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3: let a routed delete marker commit only while its latest guard holds
createDeleteMarkerIf passes an optional condition, evaluated on the
.versions directory entry under the object's lock on the owner filer,
and reports a refused commit as errLatestVersionChanged.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3 lifecycle: write the expiration marker as a compare-and-set on the latest pointer
A timed-out attempt can still be running on the S3 server while the
worker's retry writes the marker. The marker commit is now conditioned
on the .versions pointer as read just before it and skipped when the
latest is already a delete marker, so whichever attempt commits second
writes nothing.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3 lifecycle: resolve the marker as stale when another version became latest
The marker's compare-and-set only covered changes after its own pointer
read, so a version promoted after the expiration checks passed could be
hidden by the marker. The enabled path now passes the validated version
and a conflict retry that finds a different named latest resolves the
request as VERSION_NOT_LATEST.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3 lifecycle: resolve a null expiration as stale once a version takes the pointer
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3 lifecycle: read the marker's latest version the way getLatestObjectVersion does
A pointer names a version only with its file name; an id alone falls back
to the null object, and the null-is-latest signal makes the null version
current. An expiration that validated a named version no longer hides a
null written by a suspended PUT, and a current null under an id-only pointer
now expires instead of resolving as stale.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3 lifecycle: hold the object write lock for an unrouted expiration marker
Off the ring createDeleteMarkerIf cannot carry the pointer guard, so a PUT
committing between the pointer read and the marker write was hidden. The
read and the write now run under the object write lock that unrouted
writers take.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3 lifecycle: share transient-error retries between walker and replay deletes (#12649)
* s3 lifecycle: retry only transient LifecycleDelete failures
dispatchWithRetry retried every RPC error except parent cancellation, so
an authentication, permission or argument error spent the whole retry
budget before surfacing. Retry only attempt timeouts and the failures
util.IsTransientError recognizes (Unavailable, ResourceExhausted, network
resets); everything else returns on the first attempt. A nil response
is now an error instead of a nil dereference.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3 lifecycle: take a delete-rate token for every LifecycleDelete attempt
processMatches waited on the shared limiter once per match, so the
retries behind it ran outside the cluster-wide deletes/sec budget. The
retry loop now waits on the limiter before each attempt.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3 lifecycle: give walker deletes the replay retry policy
WalkerDispatcher sent each LifecycleDelete once, so one dropped
connection aborted the bucket's daily walk while the same failure on the
replay path was retried. The walker now goes through deleteWithRetry and
shares its transient-error retries, per-attempt limiter wait and nil
response handling.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3 lifecycle: retry a DeadlineExceeded the server sends back
A server or proxy can end the RPC with DeadlineExceeded while the
attempt's own deadline is still running. util.IsTransientError rejects
that status, so the delete halted with retries left. It is a timed-out
attempt like the client-side one, so retry it the same way.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3 lifecycle: say walker deletes get 3 attempts in the timeout help
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* security: refuse to start when per-target server allowlists are not applied
A component that dials every server with one option ignored
<target>_allowed_server_* keys, so a config that only set those
accepted any CA-signed server.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* scaffold: say which components scope server allowlists per target
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* security: also catch per-target server allowlists set through WEED_ env vars
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* security: optionally check the dialed host against gRPC server certificates
verify_server_host under [grpc] or a [grpc.<component>] section switches
the client to certificate-and-host verification.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* scaffold: document verify_server_host
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* s3 lifecycle: add a LifecycleDelete client that fails over between S3 fronts
When the current front returns Unavailable, the client closes it, marks it
failed for the rest of the pass, picks the next front that passes
checkLifecycleReady and resends the same request there. Callers cut off by
that switch closing their connection resend on the new front. When no front
is left, every later call returns one error naming the fronts tried.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3 lifecycle: serve the pass through the failover client
Execute hands the replay drain and the walker one failover client built on
the selected front, reports each switch as job progress, and closes the
current connection when the pass ends.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3 lifecycle: keep failing over until a front serves the resent delete
The resend after a switch went straight to the new front, so a second
lost front ended the call even with a healthy third one, and the walker
has no retry of its own. Resend through the failover loop instead.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3 lifecycle: do not recheck fronts skipped when the pass started
The failover client started with no failed fronts, so losing the selected
front checked the fronts selectS3Front had already skipped again, holding
every delete for another readiness timeout per front.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* s3 lifecycle: walk checkpoints take each entry's resume position
Versions of one object share a Path, so a checkpoint set from Path after
the first version made a resume skip the rest. The ListFunc now gives
each entry the start value that continues after it, and leaves it empty
on all but an object's last version.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3 lifecycle: resume a bucket listing where the walk stopped
The filer listing skipped entries with key <= start, but its order is
not key order: each directory emits its .versions groups before its
files, and a directory's children come before a sibling like dir.txt.
Positions are now one (pass, name) frame per directory level, so a
resume seeds each listing and skips exactly what was finished. A version
group is one unit, and a skipped group still keeps its bare null
version out of the second pass.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3 lifecycle: content hash of a walk view
Hashes every non-disabled action of a view and the versioning of their
buckets, so a saved walk position can be tied to the walk it came from:
another rule set, another walk mode, or a bucket that turned versioned
gives another hash.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3 lifecycle: persist daily walk progress and resume it
A walk that failed or was cut short started over on the next pass,
listing again every bucket it had already finished. WalkBuckets now
keeps the finished buckets and each unfinished bucket's position in a
checkpoint file next to the shard cursors, one per shard set, and the
next walk of the same view and shards resumes from it. The record is
saved every 30s and when the walk stops, and cleared when a walk
completes, so the next walk is full again and sees keys added behind
the position; a clear that fails fails the walk rather than leave a
record that would skip buckets.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3 lifecycle: a stuck resumed walk starts over
A bucket that fails at the same place on every pass kept the walk from
completing, so the buckets it had finished were never walked again and
their due objects waited forever. A resumed walk that fails again
without finishing anything clears its checkpoint, and the next walk is
full. A canceled walk keeps it.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3 lifecycle: do not resume a walk checkpoint left for a week
A checkpoint is cleared only by a later walk of the same shard set, so a
set that stopped walking, such as a one-off per-shard walk, left its
record behind, and a walk of that set much later would skip buckets
finished long before. Records now carry their save time and are not
resumed after seven days; a walk that keeps retrying saves on every
attempt and stays fresh.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3 lifecycle: a failed walk without progress leaves its checkpoint alone
A pass canceled while it read the checkpoint failed the read, then
saved its own empty progress over the record it never loaded, so the
next pass started every bucket again. A failed walk that finished
nothing has nothing new to record and no longer writes; the stuck
resume reset still clears.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3 lifecycle: space out walk checkpoint saves after a failed one
The save time advanced only on success, so once the filer refused a
periodic save every following entry tried again and logged another
warning. The interval now counts from the last attempt.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3 lifecycle: a walk checkpoint save that errors may still be on disk
A save whose reply was lost left onDisk false, so a walk that then
completed skipped the clear and the next walk resumed from the stale
record instead of walking everything.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3 lifecycle: keep a stalled walk checkpoint for a day
A resumed walk that failed again without progress dropped its record
right away, so an outage spanning two attempts restarted the whole
walk. Treat the walk as stuck only once its record has not advanced
for a day.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3 lifecycle: a resumed walk counts its earlier buckets as walked
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3 lifecycle: replay a resumed recovery walk from the walk that began it
A recovery or cold-start pass rewinds replay to runNow - maxTTL. When the
walk resumed a checkpoint, its finished buckets were listed in an earlier
attempt, so an object there that was not yet due then and whose write is
now older than maxTTL was neither expired by the walk nor replayed.
The checkpoint now keeps the start of the walk that began it, WalkBuckets
returns that time (its own start for a fresh walk), and runShard floors
the recovery rewind and the cold-start replay at it. A record without a
start is not resumed.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3 lifecycle: a cold start that resumed a walk replays on the next pass
The pass's shared subscription starts at runNow - maxTTL for a shard
without a cursor, before its walk reports an earlier start, so flooring
the cold-start drain at the walk start could not deliver the older
events. Such a shard now saves its cursor at the walk start minus maxTTL
and returns, like the recovery branch, and the next pass replays the
window from that cursor.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3 lifecycle: test that a cursor below the floor opens the next subscription
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3 lifecycle: test the cold-start replay across two Run passes
A cold start whose walk resumed an earlier attempt sees an older event
only on the next pass. The test runs Run twice against a filer that
serves events past the requested floor, with a later in-window event
that would otherwise move the cursor past the older one.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* mount: fail the write when the swap file chunk cannot be created
#7957 added a nil check after NewSwapFileChunk so a failed swap file
creation returns an error instead of panicking. The result was assigned
to pageChunk, a PageChunk interface, before the check, and an interface
holding a nil *SwapFileChunk is not nil, so the check never fired:
SaveDataAt stored the nil chunk and panicked in WriteDataAt.
Check the *SwapFileChunk before converting it. The new test points the
swap directory below a regular file; on master it panics in
SwapFileChunk.WriteDataAt.
Signed-off-by: anant <170962632+antcybersec@users.noreply.github.com>
* mount: grow the file only after its write is buffered
Write set the new size, times, pending append offset and uncommitted
bytes before AddPage. When AddPage failed (now that a swap file chunk
that cannot be created returns an error), the write was reported as
failed but the larger size stayed on the handle's entry, so the next
flush published it with a hole where the data should be.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* mount: report a partly buffered write as a short write
PageWriter.AddPage fills one chunk at a time, so when a later chunk
cannot be created the earlier chunks already hold their bytes. Return
how much was buffered, and let Write grow the size and times over that
prefix and report it, instead of failing the whole write while the
prefix still reaches the next flush.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* mount: report the bytes a failed copy_file_range already buffered
A streaming copy whose page fails partway returned an error even though
earlier pages, or the start of the failing one, were already buffered and
would reach the next flush with the destination size left short. Count
the buffered bytes, grow the destination over them, and return them as a
short copy.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* mount: free the copy test's destination pages
The buffered memory chunk counts toward the process-wide memory chunk
ceiling, so repeated runs of the test pushed later ones onto the
unusable swap directory.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* mount: count only the bytes a swap chunk stored
A swap chunk whose file write fails stores a prefix or nothing, but
SaveDataAt returned no error and AddPage counted the whole span as
buffered. SaveDataAt now reports the short write, and AddPage passes the
stored count up so a write is never acknowledged past it.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* mount: fail a partly buffered writeback write
Kernel writeback ignores the count in a WRITE reply, so a short write
there drops the unbuffered rest without an error. A write flagged as
page-cache writeback now keeps its buffered prefix but returns the
error, which the kernel reports on fsync.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Signed-off-by: anant <170962632+antcybersec@users.noreply.github.com>
Co-authored-by: anant <170962632+antcybersec@users.noreply.github.com>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* install.sh: offer the vector engine on linux only
Releases build weed-vector for linux amd64/arm64 only, so --component vector
on darwin or windows downloaded a missing asset.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* install.sh: add --enterprise
Installs from seaweedfs/artifactory: weed-enterprise-<os>_<arch>[_large_disk],
weed-volume-enterprise[_large_disk], weed-worker-enterprise and
weed-vector-enterprise. The latest tag skips vfs-* releases, and the Rust
volume server is linux-only there, so --component all skips it elsewhere.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* Revert "install.sh: offer the vector engine on linux only"
Releases after 4.53 publish weed-vector for darwin and windows too
(rust-vector-platforms.yml).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* install.sh: Enterprise vector engine is linux-only
Enterprise releases publish weed-vector-enterprise for linux amd64/arm64;
open-source releases keep their darwin and windows engines.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* test: installer --enterprise asset names, platforms and latest tag
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* install.sh: Enterprise latest is the newest plain version tag
artifactory also releases previews and a dev build, and a prerelease such
as kvcache-filer-native-v0.1.0-preview.1 sits in the release list.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* README: show passing --enterprise through bash -s --
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* install.sh: the open-source weed for windows is a .zip too
Releases publish windows_amd64[_large_disk].zip, so asking for a .tar.gz
404s on every Windows host.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* install.sh: install the windows weed as weed.exe
weed-volume and weed-vector keep their .exe on Windows; weed lost it, so
cmd and PowerShell could not run the installed binary.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* install.sh: suggest weed-volume only when it was installed
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* shell: fix s3.circuitBreaker help examples and error messages
The help examples used -type count, but -type accepts Count and MB (as
its flag description says), so both examples failed with "type not
valid, only 'count' and 'bytes' are allowed", an error that names
neither accepted value. Two other errors pointed at flags that do not
exist: -max and -counts, where the flag is -values.
Use -type Count in the examples, name Count and MB in the type error,
and -values in the value errors. TestCircuitBreakerHelpExamplesRun runs
every help example and fails on master for the two -type count lines.
Signed-off-by: anant <170962632+antcybersec@users.noreply.github.com>
* shell: name the bad action and the -actions flag in s3.circuitBreaker
The unknown-action error pointed at a nonexistent -action flag and echoed
the whole -actions list instead of the action that was rejected.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Signed-off-by: anant <170962632+antcybersec@users.noreply.github.com>
Co-authored-by: anant <170962632+antcybersec@users.noreply.github.com>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* filer: report a SQL listing that fails partway through
ListDirectoryPrefixedEntries never checked rows.Err() after its rows.Next()
loop. When the result set failed partway, for example on a lost
connection, rows.Next() returned false and the listing returned the
entries read so far with a nil error, so callers saw a shorter directory
instead of an error. Check rows.Err() once the loop has run to the end,
as the scan in abstract_sql_store_scan.go already does. A listing the
callback stopped on purpose is left alone: rows.Err() would return a
canceled request context's error there.
This covers every store built on AbstractSqlStore (mysql, mysql2,
postgres, postgres2, sqlite).
Signed-off-by: anant <170962632+antcybersec@users.noreply.github.com>
* filer: wait for the canceled SQL listing to release its rows in the test
A fixed sleep let the callback return before database/sql closed the
rows, so a listing that always consulted rows.Err could still pass.
Wait, bounded, until the connection is back in the pool instead.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Signed-off-by: anant <170962632+antcybersec@users.noreply.github.com>
Co-authored-by: anant <170962632+antcybersec@users.noreply.github.com>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
The test unmarshalled the command's output into both actual and expect,
so it compared the output with itself and the expected JSON in each case
was never checked; it also skipped the first case. Compare every case's
output with its expected config as S3CircuitBreakerConfig messages (the
output includes default fields such as "enabled": false that the
expected JSON leaves out), and restore LoadConfig when the test ends.
Signed-off-by: anant <170962632+antcybersec@users.noreply.github.com>
Co-authored-by: anant <170962632+antcybersec@users.noreply.github.com>
* dailyrun: count entries read from the filer, skipped ones included
A versioned bucket's pre-pass and a resumed walk read entries without
emitting them, so a walk can read for hours with no visited entry.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3 lifecycle: count listed entries as progress work
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
A flush uploads the dirty pages without the handle lock, then commits the
entry under it. A write in between grows the entry's size, but its pages
are not in the committed chunks. Clearing dirtyMetadata then made the next
flush skip the commit, leaving a file with the new size and no chunk for
the write, which reads back as zeros.
Count page writes on the handle and clear dirtyMetadata only when none
landed since the flush started uploading.
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* vacuum: report a request dropped because another vacuum is running
The master runs one vacuum at a time. A request that arrives while
another one is running, the automatic vacuum or an earlier request, is
dropped: Topology.Vacuum logs "Vacuum is already running" and returns.
The VacuumVolume RPC and /vol/vacuum then answered as if the vacuum had
run, so volume.vacuum printed nothing and exited 0 with no volume
compacted. Where the automatic vacuum is busy for hours, an explicit
volume.vacuum -volumeId could do nothing every time, and nothing said so.
Topology.Vacuum now returns ErrVacuumRunning for a dropped request. The
RPC returns it and /vol/vacuum answers 409 with it. volume.vacuum names
the request that failed, because a list of volumes or collections is
sent as several requests and a vacuum that starts in between fails only
the later ones. Requests are still dropped, not queued, and the
automatic vacuum ignores the error as before.
The integration tests that need their /vol/vacuum request to run ask
again while the master answers 409.
Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
* docs: note the 409 answer from /vol/vacuum while another vacuum runs
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
* s3 lifecycle: report work-based job progress during a pass
The daily pass sent progress only while starting, so a healthy scan that
ran for many hours could reach the admin's 24-hour no-progress guard.
Worker heartbeats prove the worker is alive, not that this job moves.
While dailyrun.Run is in progress the handler now sends a progress update
every 5 minutes, and only when the pass's counters (entries visited,
replay events, RPC outcomes including errors) moved since the last
delivered update. An idle tick sends nothing, so a stalled pass still
expires. A failed send is retried on the next tick. The reporter is
stopped and joined before the completion is sent, so no progress can
follow the terminal result.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* admin: test that progress, not heartbeats, keeps an old job active
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3 lifecycle: label progress phase and bucket as the last started
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* s3 lifecycle: add a LifecycleReady RPC that only authorizes the caller
LifecycleReady runs the same checkAdminAuth as LifecycleDelete and reads
no bucket or object metadata, so a lifecycle worker can confirm that an S3
front accepts its internal credentials before it starts a pass.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3 lifecycle: check S3 fronts with LifecycleReady before the pass
Execute used to dial the first registered S3 front and start loading
policies and walking buckets, so a missing or mismatched
jwt.filer_signing.key surfaced only as repeated per-bucket delete
failures. It now checks each front in order with LifecycleReady, using
the credentials LifecycleDelete sends, under a 10s bound that also
follows parent cancellation. The first front that passes serves the
pass; fronts that fail are reported in the job's progress and the log.
When none passes, the job fails before any listing with an error that
names the front and says whether this worker has the key set, without
key or token material. Fronts that predate LifecycleReady are checked
with an empty LifecycleDelete, which they authorize and then reject
before any lookup.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* s3 lifecycle: collect per-pass bucket results and work counters
WalkBuckets kept only the first bucket error and logged the rest. A
RunStats shared by the walker, the walker dispatcher and the replay
drain now records every bucket's walk result (walked, failed, skipped
on cancellation), entry and object visits, replay events, and
LifecycleDelete outcomes under the dispatch_total outcome labels.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3 lifecycle: persist the last fully successful pass
Shard cursors and walk timestamps advance on partial passes as well, so
neither says when lifecycle last covered every policy. Keep that in its
own record next to the cursors, plus a worker gauge for it.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3 lifecycle: report each pass's result in the job record
The job ended with a fixed success summary, or with only the first error,
and malformed lifecycle policies were V(1) logs. The completion now
carries the pass status (complete, partial, failed, canceled), bucket
totals and failures, invalid policies, entry, object and event counts,
LifecycleDelete outcomes, the last phase and bucket, the elapsed time
and the last full pass, on failed passes as well.
Only a complete pass advances the full-pass record; a malformed policy
makes the pass partial.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* admin: keep a failed job's result in its record
Completions with Success=false dropped Result.Summary and
Result.OutputValues, so a handler could not report what a failed run
had done, which buckets failed, or how far it got. Keep them; the error
message still drives the job's message and run record.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* grafana: chart the time since the last full lifecycle pass
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3 lifecycle: a replay halted on an unresolved delete is a partial pass
BLOCKED, RETRY_LATER and exhausted transport retries stop a shard's
replay without an error, leaving its due events for a later pass. Count
those shards and report the pass as partial, so it no longer advances
the full-pass record.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3 lifecycle: a pass with throttled walks is not a full pass
WalkerInterval can skip the walk for walk-only rules (ExpirationDate,
ExpiredDeleteMarker, NewerNoncurrent) with no error. Count those shards
and keep the full-pass record for passes that evaluated every rule; the
job reports full_pass and walks_deferred.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3 lifecycle: keep the saved full-pass time when the save fails
Publishing a timestamp that never reached the filer let the gauge move
back on the next job's reload. Keep the previous value and report the
save error in the job result.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* admin: drop a failed attempt's summary when the retry reports none
Retries reuse the job record, and failed completions now keep their
result summary, so a retry that completes without a Result showed the
failed attempt's summary, and a successful one used it as its message.
Clear the summary when a completion carries no Result, as the output
values already are.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* mount: fence a dropped handle only when it holds unpublished content
dirtyMetadata is also set by chmod, xattr and fallocate, so a failed
metadata-only close refused every later write of the inode.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* mount: fence the inode before its last handle leaves the handle map
A reopen between the map removal and the fence could publish at the
stale kernel EOF.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* mount: fence a failed metadata flush only when the handle holds content
A refused synchronous flush or an abandoned writeback metadata flush of a
handle with only chmod, xattr or fallocate changes fenced the inode, so
later writes were refused until FORGET.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* mount: count a size change on an open handle as unpublished content
A truncate or extend through an open handle changes the entry's size and
chunks, but it set only dirtyMetadata. A failed close then skipped the
final-release fence, so a reopen before FORGET could publish against the
untruncated entry.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* mount: mark copied content dirty before each page of a streaming copy
A copy that failed or was cancelled after earlier pages were accepted left
dirtyContent false, so the final-release fence skipped that content.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* mount: mark copied content dirty only after a page is accepted
A copy whose first page is rejected leaves the destination unchanged and
the kernel EOF where it was, so fencing the inode on release only blocked
later writes until FORGET.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* mount: skip the server-side copy while the destination holds unpublished content
A whole-file server copy clears dirtyContent when it commits, so a page a
cancelled streaming copy left behind would lose its release fence. Fall
back to the streaming copy instead.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* mount: skip the server-side copy while the source holds unpublished content
The filer copies the stored source, so pages still pending on the source
handle would be missing from the destination. The streaming copy reads
them through the dirty pages.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* mount: check the publication fence before a server-side whole-file copy
The filer copy publishes the destination without going through the
flush-time guard, so a fenced writeback inode could still take new
content before FORGET. Fall back to the streaming copy, whose flush is
guarded.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
filer: take the listing snapshot as a claim below later and unfinished stamps
A ListEntries snapshot tells the client that every event at or below it is
already in the listing, so the mount drops those events from a directory
build and fences them with the directory floor. The snapshot was a raw
time.Now(), which an event stamp can tie on a coarse clock: on Windows a
mkdir right after a listing of its parent was stamped with the snapshot's
own timestamp, the mount took it as listed, and the new directory stayed
missing from a parent marked cached. A create stamped before the snapshot
but not yet in the store was dropped the same way.
Take the snapshot through the same in-flight claim the idle heartbeat uses:
it stays below every unfinished stamp, and every later stamp lands above it.
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* mount: guard append gaps over concurrent content merges
* mount: fence a writeback inode when its async publication is abandoned
A drain that gives up on a non-precondition error left the token clear, so a
reopened writeback handle could publish past the stored EOF.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* mount: fence a writeback inode when its async data flush fails
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* mount: fence the inode when its last handle drops uncommitted content
A synchronous release whose final flush fails destroys the handle's dirty
pages, while under writeback the kernel EOF already counts them.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* mount: start the close()/fsync commit deadline after the data upload
Flush, Fsync and the truncate path of SetAttr create the 30 s
metadataFlushTimeout context on entry and pass it to doFlush, so the
clock also runs through FlushData, the upload of the dirty pages. That
upload has its own timeouts and retries. When an assign through a stalled
filer or a slow volume server takes most of the 30 s, CreateEntry then
fails at once with "context deadline exceeded" and close() returns EIO
for data that was already stored. The deadline is meant for the commit
alone (#10102: "a healthy CreateEntry completes in well under a second").
doFlush now takes the commit timeout and starts the deadline once the
data is uploaded. close() still cannot hang on a stuck filer: the commit
keeps its 30 s bound. Release and the flush before a rename pass 0, as
before (no overall bound, only the per-call filer timeouts).
Seen in production on a training cluster: a 30 s stall of one replica's
disk, and a filer that stalled for about 30 s, each failed close() with
EIO although the upload had succeeded.
Tests: close(), fsync() and a truncate below dirty data, each after an
assign slower than the deadline (all three fail before this change), and
a filer that never answers CreateEntry still fails close() at the
deadline.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* mount: charge the flush commit deadline only for its filer calls
Starting the 30 s deadline after the data upload is not enough.
flushMetadataToFiler does more than call the filer before it commits: it
waits for the handle's locks, reads the file back and uploads it again
when its chunks are bloated (maybeMergeChunks), and uploads a manifest
once enough chunks have built up (MaybeManifestize). None of that takes
the context. When it outlasts the deadline, CreateEntry again fails at
once with "context deadline exceeded", and close() or fsync() returns an
error for data that was stored.
The deadline becomes a budget, commitBudget, charged only while a filer
call of the commit runs: CreateEntry, and the read of a changed hard
link before it. The attempts of one flush share it, and a flush is not
retried once it is spent, so a stuck filer still fails close() after
30 s. Release, the flush before a rename and the async flush pass no
budget, as before.
Seen in production: the merge of a 1.8 MiB sqlite file with 183 chunks
took 32 s because one chunk's read first tried a volume server that was
down, and the commit failed 32 microseconds after the merge finished.
Tests: a flush whose manifest upload is slower than the deadline commits
(it fails without this change), and the filer calls of one flush share
the budget.
Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
* mount: give the stalled-lookup refusal test the new doFlush argument
TestContentFlushRefusalWithStalledLookup came in from master after this
branch changed doFlush's signature, so the merged package did not build.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
The read of the stored entry ran in the create's filer callback, so a read
that hit the call deadline turned the refusal into a filer timeout, and
both retry layers resent the refused create. Read it in its own call,
bounded to one filer call.
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* util/http: stop a full-chunk read at the chunk's recorded size
weed mount shrinks a chunk's Size in place on ftruncate but leaves its
needle untouched. A view covering the whole shortened chunk is fetched
without a Range header, so the reader returned every stored byte. The
filer GET then wrote past the declared Content-Length and sent an empty
body, and the stream reader served the stale tail.
ReadUrlAsStream and the cipher path now deliver at most size bytes;
size 0 still reads the whole needle for manifest and log chunks.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3api: limit a plain chunk read to the requested size
A full-chunk read sends no Range header, so a chunk shortened by a mount
truncate streamed its whole needle into the S3 response.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* util/http: still report a failure past the chunk size
The volume server aborts a full-needle response when the needle fails
its checksum, and only at the end. Stopping at the chunk size missed
that, so a damaged copy read as healthy. Read the rest of the body and
return its error so the caller tries another copy.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* s3vectors: call the registered weed-vector processes servers
The gateway's discovery named them workers, the name of the admin
plugin workers. Also drops two comments left without a function.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* test/s3vectors: build the Enterprise engine as weed-vector
In an Enterprise checkout, make ./weed-vector builds the distributed
engine, which also serves VectorService, instead of a separate
weed-vector-worker. An open-source checkout builds weed-vector-server
as before.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* test/s3vectors: distributed mode starts two more weed-vector engines
It started weed-vector-worker, which no target builds now, and waited
for a gateway log line that no longer exists. The Enterprise
weed-vector's engines all join the placement controller.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* mount: record a hard link's looked-up names on its inode
Lookup mapped a second name of a hard-linked file to the existing inode
without adding it to the inode's names, so GetAllPaths missed it and the
final forget left it mapped to a released inode.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* mount: keep an open file publishing when a peer unlinks one of its names
A delete event from another client marked the inode's open handle deleted
even when other names still held the hard-linked file, so every later
fsync reported success without saving metadata. The delete event's entry
carries the link count the filer decided on, so an alias delete now drops
only that name, keeps the handle on a name this mount knows, and has the
next flush read the survivor's stored entry before publishing.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* mount: test peer unlinks of a hard link name held open
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* mount: leave an unlinked file out of the periodic metadata flush
flushFileMetadata did not check the handle's delete mark. A file
unlinked while open before its deferred create reached the filer was
created again by the next periodic flush, which expects no entry.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* mount: read a pending link change before a periodic metadata flush
Only flushMetadataToFiler applied linkRefresh. After a peer unlinked one
name, the periodic flush kept publishing over the old link count, so the
filer refused every one until a regular flush ran.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* mount: test the periodic flush after a peer unlinks a hard link name
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
On Windows Lance commits local tables by rename and leaves the manifest
ETag empty, so a table dropped and written again at its location reused
the old table's lease for the same version number.
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
A peer's chunk append now merges into the flush, so these tests no longer
saw the refusal they exercise. A replaced file is still refused.
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* s3: re-check a versioned DeleteObject If-Match under the object lock
A versioned or suspended DeleteObject without a versionId routed to the
owner filer after checking If-Match outside any lock, so concurrent
conditional deletes could all pass the check and each add a delete
marker. Keep conditional deletes on the lock path, which re-checks the
precondition under the object write lock, as conditional versioned PUTs
already do.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3: guard a conditional version delete on the owner filer
DeleteObject with a versionId checked If-Match before routing the
delete to the owner filer, so concurrent conditional deletes of one
version could all report success. Carry the If-Match clause into the
owner's transaction as a guard on the version (the null entry for
versionId=null), evaluated under the object's path lock before anything
applies, and map that guard failing to 412 instead of the object-lock
403. An If-Match the filer cannot evaluate stays on the lock path.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3: carry an If-Match ETag list into the routed delete
A DeleteObject whose If-Match listed several ETags fell back to the
gateway's object write lock, while a single-ETag delete of the same key
ran under the owner filer's path lock. The two locks do not exclude each
other, so a list delete and a single-ETag delete could both report
success. Send every candidate to the filer, whose strong comparison
matches checkDeleteIfMatch, so all conditional deletes with an owner are
decided under the owner's lock.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3: fail a conditional null delete on an owner transport error
The owner may have committed the delete before the error, and the gateway
lock path it fell back to does not exclude routed deletes, so a second
If-Match delete could pass there too. Answer 500 as the specific-version
path does; unconditional deletes still fall back.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3: fail a conditional null delete on any owner error but the nested refusal
Only the owner's non-empty-folder refusal is known to apply nothing. After
any other owner error the delete may have been partly applied, and a lock-path
re-check does not exclude a concurrent routed delete of the same version, so a
conditional null-version delete answers 500 instead of falling back.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3: strip a nested conditional null delete on the owner
A conditional versionId=null delete that the owner refuses because other
keys are nested under the entry used to fall back to the gateway lock and
re-check If-Match there. That lock does not exclude routed deletes: once
the last nested key is removed, a concurrent routed delete can pass the
same check on the owner, and the fallback then treats the missing entry as
deleted, so both requests answer 204.
Demote the entry on the owner instead, in a transaction guarded by the
If-Match clauses and by the entry it was read as, with the object-lock
guards on the request. A missing or already demoted entry answers 412.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* filer: add an IF_OBJECT_EXISTS write condition
IF_EXISTS also holds for a plain directory, which is what is left of a key
once the object on it is stripped while other keys stay nested under it.
IF_OBJECT_EXISTS holds only for an entry that carries an object.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3: require an object for a conditional null delete with If-Match: *
Stripping the null version off an entry other keys are nested under leaves a
plain directory. Once the last nested key is gone, a routed delete with
If-Match: * still found an entry there and removed the empty directory, so a
second conditional delete could also answer 204. The owner now checks that
the entry carries an object.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3: settle a wildcard null delete on the entry when the filer predates IF_OBJECT_EXISTS
A filer older than IF_OBJECT_EXISTS fails it closed, so with the gateway
upgraded first every If-Match: * delete of a null version answered 412. When
the narrowed guard fails, the gateway now reads the entry on the owner: absent
or a plain directory is a 412, otherwise the file is deleted, or the directory
stripped, under IF_EXISTS and IF_ENTRY_EQUAL on the entry it read. The demote
keeps the caller's If-Match clauses, since the entry guard already requires an
object.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* filer: cancel a chunk download that no reader waits for
* filer: stop waiting for reader cache budget once cancelled
* filer: keep a cancelled download's re-lookups off its context
The refresh and retry lookups took the download's context. FilerClient
coalesces concurrent lookups of a volume, so a cancelled download leading
one failed the other readers waiting on it. Fetches still stop on cancel.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* filer: test that cancelling a download leaves its re-lookup running
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* filer: cancel unheld downloads when the reader cache is destroyed
ReaderCache.destroy waited for every downloader, so closing a reader
waited for a stalled prefetch nobody read. Downloads with no reader or
stream on them are now cancelled first; ones still in use keep running.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* filer: test that destroying the reader cache cancels an unread prefetch
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* filer: cancel a destroyed download once its last holder leaves
destroy skipped downloads a reader or stream still held, and once they
were out of the map nothing cancelled them, so destroy waited for a fetch
nobody used. removeConsumedLocked now cancels any consumable download,
whether or not it is still in the map.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* filer: test a destroyed download cancelled after its last stream leaves
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* filer: release the stream only after destroy dropped the download
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* filer: test a cancelled stream leaving a waiting reader's download running
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* s3tables: re-anchor legacy owners for federated STS sessions
A web-identity role session was keyed on its bare OIDC subject until
owners became issuer-scoped. Its sessions now carry the scoped aws:userid
but no asserted issuers, so the legacy subject check never applied and the
session lost the table buckets and namespaces it had created. Read the
configured issuer set for such sessions.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3api: keep token claims from asserting trusted issuers on STS sessions
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* iceberg: give the catalog's table handler the local accounts and issuer set
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3tables: test a federated session on a bucket its legacy subject owns
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* sts: drop a token-supplied aws:userid when a session has no parent user
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3api: take a session's aws:userid from its signed parent user
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3api: test that session claims keep only the server-asserted aws:userid
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3tables: match legacy subject patterns in table bucket policy principals
A policy written before issuer scoping can name role sessions by a pattern
over their bare OIDC subjects. Match such a pattern against the caller's
subject claim once that subject resolves to the session's scoped principal
under the single configured issuer.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3tables: test a federated session against a legacy subject pattern principal
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* s3api: refuse a PUT to a stored version's physical path
In a Suspended bucket a PUT to <key>.versions/v_<id> replaced the stored
version's body in place. Look the entry up when the key has the version
file shape and deny the write when it is a version file.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3api: refuse copy, multipart, POST and rename writes to a version file
Apply the same version file check to the CopyObject, CreateMultipartUpload,
CompleteMultipartUpload, POST Object and RenameObject destinations.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3api: read a version file write target on its logical owner
The guard resolved the key as an object, which on a ring reads the
physical path's owner and, in a versioned bucket, may return a nested
version under <key>.versions/v_<id>.versions instead of the stored
version. Look up the physical entry directly, routed by the logical key.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3api: confirm a version file miss on a skipped owner
When the owner was recently unreachable the routed lookup answers from a
replica, whose miss does not show the version is absent. Ask the owner
directly in that case and refuse the write when it cannot answer.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3api: confirm every routed version file miss on the owner
Whether the routed lookup skipped the owner can change between the lookup
and a second unreachable check, letting a replica's miss through. Ask the
owner after any miss instead.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* fix(mount): retain refused async metadata before releasing handle
* mount: sync the failed-metadata directories with util.FsyncDir
Directory fsync fails on Windows, where the mount also builds, and returns
EINVAL on some filesystems. Either kept the retention loop retrying a record
that could never be saved, so the drain never finished.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* mount: keep failed-metadata records in a per-user directory
The write cache root defaults to the shared temp directory. The first mount
to save a record created failed-metadata there as 0700, so another user's
mount could never save one and its drain retried forever.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* mount: drop the async conflict probe test
TestContentAsyncConflictKeepsRecoveryRecord drives the same refused drain
with the retry backoff stubbed out; the probe slept through the real backoff
(7 s) and its kept-handle branch could not run. Its peer-entry check moves
into the recovery record test.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* filer/etcd: client keepalive and per-request deadlines
An etcd member that goes silent with its connections open (a dead or hung
node, a network partition, a frozen process) keeps the store's requests
balanced onto it waiting forever, because the client has no keepalive and
the calls have no deadline. Writes through the filer stop although the
other members keep quorum.
keepalive_timeout (default 3s, "0s" turns keepalive off) enables gRPC
keepalive with the ping interval fixed at gRPC's 10s minimum. gRPC also
sets TCP_USER_TIMEOUT to it on Linux, so a connection whose sent data goes
unacknowledged that long is closed by the kernel. request_timeout (default
3s) bounds every Get, Put and Delete unless the caller set a deadline;
timeout keeps its meaning as the dial timeout. DeleteFolderChildren gets no
request deadline: its DeleteRange grows with the directory, and keepalive
still ends it if the member goes silent.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* Update weed/filer/etcd/etcd_store.go
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
* Update weed/filer/etcd/etcd_store_test.go
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
* Update weed/command/scaffold/filer.toml
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
* filer/etcd: no request deadline on entry writes
A Put whose client deadline expires may still have been committed. The
filer's upload handler treats the error as a failed write and deletes the
uploaded chunks, which the saved entry may then point at. InsertEntry and
UpdateEntry therefore keep the caller's context: etcd's own request
timeout ends a Put a live member cannot commit, and keepalive ends one on
a member that goes silent. Reads and deletes keep request_timeout; a
timed-out store delete returns before the filer deletes any chunks.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* filer/etcd: no request deadline on KvPut
The filer wrapper stores a hard-linked file's attributes and chunk list
through KvPut. An HTTP append to a hard-linked file whose KvPut times out
after etcd saved it returns an error, and the upload handler deletes the
appended chunks the saved hard link now points at. KvPut keeps the
caller's context like InsertEntry and UpdateEntry.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
When a mount is rooted at an existing directory, two code paths replace
that directory's filer entry with one the mount builds itself:
- every mount start re-creates the root with a non-exclusive mkdir,
which the filer handles as an update of the whole entry;
- creating or removing an entry directly in the root saves the root's
entry to touch its mtime, and maybeLoadEntry synthesizes that entry
from the mount options instead of reading the filer.
The filer keeps only crtime and inode on such an update, so the root
loses its mode, owner and extended attributes. When the root is a
bucket it also loses the bucket quota, which silently drops to 0
(unlimited), and the settings S3 keeps on the bucket entry. The explicit
allowEmptyFolders=false that #11370 meant to keep is replaced by true at
every mount start, because the mkdir has wiped it before
ensureBucketAllowEmptyFolders looks.
Create the root only when a lookup says it is missing, and then
exclusively. Leave the root out of the filer-side mtime touch: its
attributes come from the mount options (setRootAttr), so the filer copy
of its mtime was never visible through the mount anyway.
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* mount: keep flushing an open file whose other hard link survives an unlink
Unlink marks the inode's open handle deleted so no flush recreates the
removed name. When the entry was one of several hard links the inode lives
on under another name, yet the mark stayed and every later fsync and close
returned OK without publishing metadata. Lift the unlink's own mark once
the name is gone and the inode still has a path.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* mount: carry a link's stored entry into the file's open handle
Link rewrites the source entry with the new hard link id, count and times,
but an open handle kept its old base and entry. Its next conditional
content flush was refused (EIO), and an unconditional one would have
dropped the hard link id. Move the handle onto the stored entry when only
the link fields and times changed.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* mount: carry an alias unlink's stored entry into the open handle
Removing one of several hard links lowers the stored link count and
stamps a new ctime, and the handle may still hold the removed name. Look up
the entry at the name the handle flushes through and adopt it, so the next
content flush matches the stored entry, writes the right count, and
attribute saves address the surviving name.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* mount: test an open handle across link and alias unlink
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* mount: read an alias unlink's lost entry again at the next flush
When the read of the surviving entry right after an alias unlink fails,
the handle kept a base with the old link count and ctime, and every later
flush was refused even after the filer recovered. Leave the hard link id
on the handle and have the next flush read the stored entry and adopt it
before publishing.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* mount: publish a queued close before unlinking a name another link keeps
With writeback caching, close only queues the flush. Unlink marked the
handle deleted before waiting for that flush, so the flush skipped its
metadata and the surviving link kept the content from before the close.
When another link keeps the file, wait for the queued flush first.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* mount: let a queued close reach the filer before linking the file
With writeback caching a new file is created on the filer by its queued
close, and a listed directory serves the source from the cache, so a link
right after close updated a name the filer did not have yet and failed
with EIO.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* mount: skip a link change no newer than the handle's entry
A link change carried into the handle could replace a base already moved
past it by a later acknowledgment. Use the same clock-domain check as
installAckedEntry.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* mount: keep a link change's ctime in the dirty entry
The flush does not restamp times, so a dirty entry carrying an earlier
write's ctime moved the filer's ctime back from the link or unlink.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* mount: flush through the file's current name when the resolved one is gone
doFlush resolves the name before uploading the data. An unlink of that
alias, or a rename, in the meantime sent the publish to a name the file no
longer has.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* mount: keep the pending link read when its name is already gone
A flush could read the removed alias name, find nothing, and drop the
pending read of the surviving link, leaving the stale base in place.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* mount: test link and alias unlink across queued closes, flush races, versions and ctime
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* mount: pick a flush's name again once its locks are held
An unlink or rename that lands between the name pick and the locks would
otherwise send the flush to the removed name, where the create guard fails.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* mount: test a flush waiting on its lock through an alias unlink
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* mount: check the flush target after every lock acquisition
A flush whose name kept changing published to the name chosen before its
last lock wait. Check after each acquisition and return a retryable error
once the attempts run out.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* mount: test a flush whose name changes during every lock wait
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* shell: return flag parse errors from every command
48 commands returned nil when their flags failed to parse, so a mistyped
flag printed the flag error but the command still counted as a success:
a piped weed shell run exited 0, the master maintenance loop logged no
error, and admin script jobs reported the line as done.
Return the parse error as s3.bucket.create (#12544) and the other shell
commands already do. TestCommandsReturnFlagParseErrors runs every
registered command with an unknown flag and fails for the 48 on master.
Signed-off-by: anant <170962632+antcybersec@users.noreply.github.com>
* shell: don't report -h as a failure
With the parse errors returned, -h on those commands now returns
flag.ErrHelp, so the shell printed "error: flag: help requested" and a
piped run exited 1. The flag package has already printed the usage, so
processEachCmd now treats flag.ErrHelp as success for every command.
Signed-off-by: anant <170962632+antcybersec@users.noreply.github.com>
* shell: keep the -h test beside the flag parse test
TestProcessEachCmdTreatsHelpAsSuccess covers the same change as
TestCommandsReturnFlagParseErrors, so it lives in the same new file and
shell_liner_test.go stays as it was.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Signed-off-by: anant <170962632+antcybersec@users.noreply.github.com>
Co-authored-by: anant <170962632+antcybersec@users.noreply.github.com>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* credential: migrate the legacy IAM file without overwriting newer entries
The migration wrote every identity, service account and group from the
snapshot the load read, over whatever the per-entry files held by then, so
a revoke or delete landing in between was undone for good. It now migrates
only the legacy file's own entries, creates each per-entry file only when
none exists, and waits for the next load when the legacy file changed
after the snapshot.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* credential: test changes landing between the IAM load and its migration
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* credential: keep a per-entry file deleted after the IAM load deleted
An entry the load read from its own file is already migrated. When that
file is removed before the migration runs, the legacy copy must not
recreate it.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* credential: test a deleted per-entry file the legacy file also holds
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* s3tables: create a missing directory exclusively in ensureDirectory
ensureDirectory looked the path up and, when it was missing, created it with a
plain CreateEntry. A table create that lost the lookup race then replaced the
directory another create had made, claimed and confirmed since, erasing that
table's record, so a third create could claim the name and win as well.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* iceberg: create missing metadata parent directories exclusively
saveMetadataBlob created each missing directory on the way to the metadata
directory with a plain CreateEntry after a lookup. A table directory another
create had made and written its record onto since the lookup was replaced by a
bare directory, erasing that record.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* iceberg: create missing stage-create marker directories exclusively
The marker path starts at the table bucket directory, which carries the bucket
record; a plain create after a missed lookup replaces a directory another writer
made in between, with whatever it carries.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Volume layouts are keyed by ttl.String(), so a nil ttl and the empty ttl
share one layout. ttlEqual told them apart, so a replica reported with a nil
ttl into a layout created with the empty ttl matched its own layout as a
duplicate, and every copy was quarantined.
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* shell: replicate the downloaded volume before deleting its remote object
volume.tier.download downloaded every replica from the remote tier and let
the last one in the master's location list delete the shared object, so a
replica missing from that list kept reading a deleted object.
Download onto one replica, copy it over every other remote-backed replica,
re-read the topology so replicas that show up meanwhile are replaced too,
and only then delete the last remote-backed replica together with the
object before copying the local volume onto it.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* shell: test tier download replicates before deleting the remote object
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* shell: copy the downloaded volume onto each replica's own disk type
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* shell: test tier download keeps each replica's disk type
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* shell: name the hdd disk type when copying a downloaded volume
VolumeCopy treats an empty disk type as the source's, so an HDD replica
copied from an SSD source was placed on SSD or failed.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* shell: block deletes on the tier download copy source
A readonly-can-delete source took deletes after VolumeCopy read its
index, so the replaced replica lost them and served the files again.
Hard-mark the source readonly while replicas are replaced, then restore
the writable mark or reload the persisted state.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* shell: keep the remote object while replicas are unlisted
Tier download deleted the shared object once the listed replicas were
local, even when the replica placement expected more copies than the
topology listed. Keep it in that case so a rerun can finish later.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* shell: fail tier download when no replica is listed
Also point the final-replica delete error at volume.fix.replication.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* shell: keep the remote object when a server lists the volume twice
A stale twin on another disk of a replaced server still references the
object but is filtered out with its server.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* shell: restore the tier download source when its readonly mark fails
VolumeMarkReadonly blocks local writes and deletes before its second master
notification, so a failed notification returns an error with the source
already frozen. Restore it whenever the status was read.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* shell: report a failed tier download source restore
A restore that leaves the source unmounted or frozen was only printed, so the
command still returned success. Join the restore error into the result.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* shell: wrap the tier download restore error
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* volume_move: persist the move's source freeze across reloads
A live move froze the source in memory only, so an unmount/mount or a
volume server restart mid-move reopened it for writes that the move then
deletes with the source. Persist the freeze when the move froze a
writable source, and reopen the target right after the copy since the
copied .vif carries the freeze. An already-readonly source keeps an
in-memory mark, and a failed move still restores the source writable.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* test: live move keeps the source frozen across a reload, on Go and Rust volume servers
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* test: require 409 from the reloaded frozen source
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* volume_move: reopen an interrupted move's source only without a target copy
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* mount: merge a refused content flush onto the stored entry
A content flush conditioned on the handle's base failed with EIO whenever a
peer had changed the file since, so two mounts appending to one file could
not both close it cleanly. On a refused condition, reread the entry and
replay this handle's changes since its base onto it: the chunks it added or
dropped, and the attributes, extended keys and inline content it changed.
A local truncation keeps its size and clips the peer's chunks to it;
otherwise the larger size wins. The merged entry is published against the
stored one, up to three times, and installed in the handle so its next
flush keeps the peer's chunks. A replaced or deleted file is still refused.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* mount: merge a peer's content change into a dirty handle
A subscription event carrying a peer's content change replaced a dirty
handle's entry and discarded its unflushed pages, so the handle's writes
were lost and its close still returned OK. Fold the peer's entry into the
handle with the same merge a refused flush uses, keep the dirty pages, and
take the peer's entry as the new base. A replaced file still invalidates.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* mount: stop a flush merge once a filer accepts the write
withFilerClient can call the callback on several filers. A refusal from
one left its stored entry behind, so a later filer's success went on to
another merge round.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* mount: refuse a merge that puts inline content next to chunks
A read inside the inline bytes returns them without looking at chunks,
so one side's inline content merged with the other side's chunks hid
data after a successful flush.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* mount: refuse to merge a whole-file rewrite over a peer's chunk change
maybeMergeChunks re-uploads the handle's whole view as unstamped chunks.
Merged after a peer's change, they tie with a peer's own rewrite or bring
back a range the peer truncated, so that flush fails as before.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* mount: clip peer chunks at the lowest truncate, not the final size
A handle that truncated and then wrote past the old end ends up larger
than its base, so the merge did not see the truncate and kept the peer's
chunks inside the range the truncate cleared. SetAttr now records the
lowest size truncated to over the current base.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* mount: keep a peer's truncate when the handle left the size alone
The merge took the larger of the handle's and the stored size, so a peer
truncate came back as a zero-filled tail when the handle only overwrote
bytes inside it. The stored size now stands unless the handle changed
its size or wrote past it, and rebasing keeps only the growth writes made
after the flush was taken.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* mount: retry a read view whose merged manifests did not resolve
A failed SetChunks leaves the chunk group answering every read with the
resolution error, and nothing re-ran it while the entry stayed the same.
The handle now marks such a view (on rebase and on SetEntry) and the
next read or lseek resolves the entry's chunks again.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* mount: keep a dirty handle's writes when a peer change cannot merge
The new merge refusals sent a peer event on the same file down the old
path, which discarded the dirty pages and adopted the peer's entry, so
close returned OK without the writes. The handle now keeps its writes and
base, and the flush meets the conflict and fails.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* mount: keep an unpublished truncate across a peer's base change
The truncate mark was tied to the base pointer, so a peer event that moved
a dirty handle's base (a merged content change or a metadata-only change)
dropped it, and a later merge on a regrown file kept peer chunks inside
the truncated range. Those base moves now carry the mark; a flush ack or a
discarded handle state still clears it.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* mount: wait for the flush before merging a shrink over buffered writes
A write inside the file leaves its bytes in dirty pages without moving
the size. Merging a peer's truncate then adopted the smaller size, and the
chunk uploaded later sat past it, unreadable through the handle. The event
path and the periodic metadata flush now refuse a merge that shrinks the
handle; the close flush uploads the pages first and merges then.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* mount: tell a shortened base chunk from a whole-file rewrite
A truncate shortens a base chunk in place, so it gets a new reference
key, and an old chunk can carry no timestamp. The rewrite refusal now
applies only to unstamped chunks whose file id is new to the base.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* mount: check the creation time when the base has no inode
An entry without an inode or hard link id matched any replacement at the
same path, so a stale handle could merge into a different file. Without
an inode the merge now needs the same creation time. The filer keeps it
across updates; only a synthesized base has neither and still merges.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* mount: keep a dirty handle's writes when a peer replaces the file
A peer entry with another inode at the handle's path went down the
fallback that destroys the dirty pages and installs the replacement, so
close returned OK without the writes. The handle now keeps its writes
and base like it does for an unmergeable change, and its flush refuses to
update the replacement.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* mount: build the content flush tests on freebsd
The invalidation tests, which every platform compiles, now use the
content flush helpers, so the freebsd cross-compile vet failed with
undefined: flushConditionEntry. Nothing in the file is freebsd-specific.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* mount: keep a dirty handle's base when a same-content file replaces it
A replacement with the base's chunks but another inode counted as a
metadata-only change, so the dirty handle adopted it as its base and the
next conditional flush overwrote the new file.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* mount: bring a peer's metadata change into a dirty handle
A metadata-only event moved a dirty handle's base but left its live
entry, so the next flush read the peer's chmod as a local edit and
published the old mode back. Replay the handle's own changes onto the
peer's entry instead, as for a content change.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* mount: wait for the flush before merging peer inline content over buffered writes
A buffered write is not a chunk yet, so a merge could install a peer's
inline content that later reads return ahead of the write's chunk.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* mount: track a whole-file chunk rewrite on the handle
The merge told a rewrite by its unstamped chunks, which a manifest fold
hides: manifests are unstamped either way. Record the rewrite against the
base it was made from, carried across peer base changes like a pending
truncate, and refuse the merge from that.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* mount: refuse a merge that shortens a manifest at the truncate
Clipping a manifest reference leaves its members whole, so peer bytes past
the handle's truncate could reappear once the file grew again.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* mount: flush a dirty handle after a peer replaces its file
The replacement test checked only the retained handle state. It now
flushes against a filer holding the replacement, including one with the
base's content, and checks the flush fails and leaves it unchanged.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* mount: test an append fsync after a peer's attribute-only change
A second mount's acknowledged chmod, chown, utimes or setxattr leaves
the content alone, so the writer's conditional content flush must still
publish its append and keep that change. On master the refused flush
surfaces as EIO from Fsync.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* s3api: sign every operation subresource in signature V2
The V2 canonical resource only kept the subresources inherited from the
original allowlist, so tagging, retention, legal-hold, cors, encryption,
object-lock and the other keys the router dispatches on were left out of
the signature. A V2 signature issued for one request therefore also
verified that request with one of those keys appended, running a
different operation under the signer's identity.
Add every operation-selecting subresource to the signed list. Clients
that already sign tagging, cors, object-lock or replication in V2 now
verify as well.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3api: sign every value of a repeated signature V2 subresource
canonicalizedResourceV2 kept only the last value of a repeated query key,
while handlers read the first. A request signed for versionId=A could be
sent as versionId=B&versionId=A and act on version B. Sign every value in
order, as V2 clients do, so a second value no longer verifies.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* fix(mount): preserve acknowledged chunks during attribute updates
* mount: keep the WFS field block aligned
Adding the publication-lock fields inside the aligned block made gofmt
re-indent 25 unrelated lines. Put them in their own group after
dirIdleEvict, matching the enterprise layout, so syncs do not conflict there.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* mount: finish attribute saves through a FUSE interrupt
saveEntryAttributes derived its RPC context from the syscall's cancel
channel, so an interrupt while the conditional UpdateEntry was at the
owner returned EIO for an update the owner then applied, and skipped the
local install and retention. Keep the metadata deadline but drop the
cancellation, like the other metadata mutations.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* mount: map attribute CAS ids in place instead of cloning the entry
saveEntryAttributes cloned the whole entry three times per attempt only
to rewrite uid/gid for the filer, and each clone is as large as the
chunk list. Switch the ids in place and restore the exact local values
after the call, keeping one clone. The test now checks the reply carries
the local uid.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* mount: compare attribute updates against the filer's own uid/gid
With -map.uid/-map.gid, a filer id that is also a mapped local id reads
back unchanged and maps forward to another filer id. The IF_ENTRY_EQUAL
expected entry then never matched, so chmod, utimes and xattr changes on
such files failed with EAGAIN after three attempts, and an update would
have rewritten the owner. Keep the filer ids read with the entry for the
ids the syscall leaves alone.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* fix(mount): align attribute retention with read lock order
* mount: take the handle lock before entryLock in the attribute snapshot
attributeEntrySnapshot held entryLock while it queued for the shared
handle lock. A Read holding that lock waits on entryLock, the snapshot
waits behind a queued exclusive (an invalidation or a Write), and the
exclusive waits on the Read.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* fix(mount): complete handle lock order and preserve explicit ID intent
* mount: re-read a refused attribute update from the file's owner filer
The filer checks the IF_ENTRY_EQUAL condition on the owner of the path's
write route key. The retry read any filer, so behind one that had not
applied the owner's latest update, every attempt sent the same stale base
and the syscall failed with EAGAIN. Read from the owner once the lock ring
is known, like lookupFilerEntry, but keep the filer's raw ids.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* fix(mount): condition content flushes on the observed entry
* mount: stop retrying a flush the filer refused on its condition
The filer reports a failed CreateEntry condition as FilerError_PRECONDITION_FAILED
in the response, which surfaces as filer_pb.ErrPreconditionFailed rather than a
FailedPrecondition status. The flush retry loop treated it as transient and
replayed the refused flush three more times over 7 s before returning EIO.
The test filer now answers like the real one.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* mount: take the handle base from the entry the filer stored
After an acknowledged flush the handle base was the requested entry, but the
filer stores it with a fresh commit_ts_ns (and fills crtime, inode, atime), so
the next IF_ENTRY_EQUAL flush on the same handle never matched: write, fsync,
write, fsync returned EIO, as did close after a periodic metadata flush and the
first flush after an eager create. Use the stored entry from the
acknowledgement's metadata event.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* mount: condition the periodic metadata flush like the close flush
The periodic metadata flush of an open, dirty handle sent the same whole entry
through an unconditional CreateEntry, so a file held open past the flush
interval still overwrote a peer's acknowledged append, and the close flush then
matched the overwritten state. Both paths now publish through one helper that
carries the handle's write condition.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* mount: republish a refused flush once when the stored entry is unchanged
A handle's base can differ from the stored entry without any peer write: a
rename while the file is open restamps the entry under its new name, and with
-map.uid a filer id that is also a mapped local id does not map back to itself.
Each made every flush of the handle fail with EIO. On a miss, read the stored
entry and, if it still holds the observed state apart from name, commit stamp,
id encoding and derived size, retry once with that entry as the condition.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* mount: take the filer's entry once a write open holds the DLM lock
The lock orders this writer after the previous holder, but the holder's
flush may not have reached the local cache yet, so the handle's base
named the entry from before the wait and the conditional flush refused.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* mount: share the open-existing path of Create
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* mount: open the file a previous DLM holder created during a deferred create
A deferred create never asks the filer, so a create that waited for the
lock went on to publish over a file the previous holder had just made,
and its flush expecting no entry was refused.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* master: send the lock ring snapshot to mounts
A mount running -dlm is a lock client like the S3 gateway, and needs the
ring on connect to reach a file's owner filer before the next membership
change is broadcast.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* mount: follow the masters' lock ring under -dlm
The lock client only knew the seed filer, so every lock took the forward
hop and the mount had no way to tell which filer owns a file. Subscribe to
the masters named by the filer, like the S3 gateway, and wait briefly for
the first ring before serving.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* mount: read the DLM-ordered entry from the file's owner filer
The filer checks a flush's condition on the owner of the path's write
route key, and another filer may not have applied the previous holder's
flush yet. Look the entry up on that owner once the ring is known.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* volume(rust): untrack queued writes before replying
The durable write worker sent each write's result and only then dropped
the batch's pending-write trackers, so a caller could see its write
finish while the write was still tracked as pending. Go untracks in the
caller before writeNeedle2 returns. Drop the trackers before sending the
results so the Rust queue path matches.
test_replicate_delete_cancels_a_queued_write asserted the count was zero
right after the fsync write returned and failed about 1 run in 100 under
load (Rust Unit Tests on master c2625dd).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* volume(rust): untrack a write rejected for a remounted volume before replying
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* volume: skip VolumeDelete when its RPC was cancelled before the delete
A caller that cancels (a lost tiering lease, an expired deadline) still
had its admitted delete run in the Go server, including one that waited
behind the copy guard. The Rust server drops a cancelled handler, so
return the context error before the delete starts.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* volume: wait until the queued delete blocks on the guard before cancelling
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* s3api: verify signed request bodies against the signed payload hash
A SigV4 signature covers the declared x-amz-content-sha256, but only
getRequestDataReader (PutObject, UploadPart) compared the body with it.
Every other handler parsed the body as received, so a signed request,
DeleteObjects for example, could be replayed with a different body.
After a V4 signature verifies, wrap the body so it is handed out only
once all of it has been read and matched against the hash the signature
covered; a decoder that stops reading early never sees unverified bytes.
getRequestDataReader unwraps it and keeps verifying uploads as they
stream.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3api: answer XAmzContentSHA256Mismatch for refused DeleteObjects bodies
DeleteObjects answered a body that failed its payload-hash check with
InternalError and CompleteMultipartUpload with MalformedXML. Name the
check that failed instead, as PutObject does.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3api: check Content-MD5 on signed request bodies
Signature V2 signs Content-MD5 rather than a payload hash, and a V4
request sent with UNSIGNED-PAYLOAD may still sign it, but bodies read
outside the upload path never compared it with the body. Check it with
the payload hash once a V2 or V4 signature verifies: BadDigest on a
mismatch, InvalidDigest when it is malformed.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3api: verify presigned uploads against a signed payload hash
A presigned V4 URL can sign X-Amz-Content-Sha256 as a query parameter,
but getRequestDataReader compared upload bodies only with the header of
that name, so the signed hash went unchecked. Verify uploads against the
hash the signature covered.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3api: check Content-MD5 on bodies buffered for their payload hash
IAM and other non-S3 requests signed without x-amz-content-sha256 have
their body buffered and hashed during signature verification. That
buffer counted as already bound, so their Content-MD5 was never checked.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3api: check Content-MD5 on directory marker bodies
A PUT of a key ending in "/" stores its body on the directory entry
without going through putToFiler, so its Content-MD5 was never compared
with the body. Answer BadDigest on a mismatch, as putToFiler does.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3api: wrap and unwrap signed bodies through helpers
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* shell: confirm trim survivors still serve the volume before deleting
volume.fix.replication compared the doomed replica's index with every
survivor and then deleted it unconditionally. A survivor that went away
while the indexes were read still let the delete through, leaving fewer
copies than the check had counted. Re-read each survivor's VolumeStatus
after the index checks and skip the trim if any no longer has the volume.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* shell: test a trim whose survivor goes away after its index read
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* s3: ignore unsigned object headers on POST Object
The POST policy signs form fields only, but the upload read tagging,
user metadata, storage class, object lock and the stored content
headers from the request headers as well. Drop them before the form
fields are applied.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3: test POST Object request headers against signed form fields
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* s3api: answer PutObjectLegalHold on a missing key with NoSuchKey
In a versioned bucket a key with no latest version resolves to
ErrLatestVersionNotFound, which the legal hold handler did not map, so
the request failed with 500 InternalError. PutObjectRetention already
maps it to NoSuchKey.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3api: keep failed reads distinct when resolving an object lock target
getObjectEntryForUpdate turned every lookup error into a not-found
sentinel, so a failed filer read made PutObjectRetention and
PutObjectLegalHold answer NoSuchKey for an object that exists. Only a
genuine miss becomes the sentinel now, matching getObjectEntry.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3api: answer object lock calls on a delete marker with NoSuchKey
PutObjectLegalHold and PutObjectRetention on a key whose current
version is a delete marker returned 200 and wrote the lock onto the
marker, and the matching GETs read it back. A key whose current version
is a delete marker does not exist, and delete markers take no lock, so
these calls now answer NoSuchKey like the tagging and ACL handlers.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3api: let a lock recorded on a delete marker not block its removal
Delete markers are not WORM-protected, but a legal hold or retention
written onto one (possible before the previous change) made both the
gateway check and the routed delete's filer-side WORM guard refuse to
remove the marker, so the object could not be restored and lifecycle
could not clean the expired marker up.
Lock enforcement now ignores a lock on a delete marker, and the routed
version deletes in DeleteObject and lifecycle drop the WORM guard when
the version they read is a marker. A version id never changes kind, so
the marker read is the one deleted; the null version keeps its guard
because a PUT can replace it.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
The master, volume, volume-rust, server, mini, filer and s3 cases and the
fallback forwarded the user arguments as an unquoted $@, so the shell split
each one at whitespace and globbed it against /data. An -admin.password with
a space reached weed as two words, and weed ignored every flag after the
stray word. Quote "$@" as the other cases already do; $ARGS stays unquoted
since the script builds it from its own words.
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* replication: let the local sink check a delete against the live source
LocalSink removed the destination path for any delete it was handed. A
delete that reaches the sink after the source recreated the path (a peer
filer's event delivered late, or a historical event replayed) removed the
newer file, and no later event brings it back.
DeleteEntryVersion receives the deleted entry. Before removing an existing
destination, the local sink looks up the source path and keeps the
destination when the source holds an entry of the same kind committed
after the deleted version (commit_ts_ns, or mtime for unstamped entries).
A live entry that is the deleted version itself or older means the source
store lags the event, so the delete proceeds. A failed lookup returns the
error so the event is retried instead of deleting a file that may be live.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* filer.sync, filer.replicate: hand the deleted version to the sink
Pure deletes, renames without a native move and moves out of the
watched directory now go through sink.DeleteEntryVersion with the old
entry, so a sink that checks the live source (the local sink) can keep a
path the source recreated after the deleted version. Other sinks fall
back to DeleteEntry unchanged.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* replication: order the local sink check by the delete's mutation time
A source filer can hold a version written concurrently with the delete:
newer than the deleted entry but older than the delete itself. The
source's own replay applies the delete over it, so keeping the backup
copy because it outranks the deleted entry left a file the source no
longer has.
The sink now gets the delete's mutation time (MutationTsNs, else TsNs)
and keeps the destination only when the live entry was committed after
it, the order Filer.Replay keeps. That also covers a stamped replacement
of an unstamped entry whose mtime did not move. filer.replicate events
carry no time, so they keep the deleted entry's stamp as the bound, and
mtime only when nothing is stamped.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* replication: count an unstamped live entry from its mtime
A source filer that predates commit stamps stores entries with no
CommitTsNs, so comparing the live stamp with the delete's time never
kept a replacement there. An unstamped live entry now counts from its
mtime against the delete's time.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* replication: keep a stamped live entry when the deleted one is unstamped and the delete has no time
filer.replicate gets no delete time from the notification queue. When the
deleted entry has no commit stamp either, a stamped live entry was written
after it, so its mtime no longer decides.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
The line parser only trimmed quotes, so `key = "abc\n"` gave the Rust
volume server the bytes `abc\n` with a literal backslash while the Go
servers sign with a newline, and every JWT failed. Literal strings kept
their quotes and a trailing comment stayed part of the value.
Decode each value with the TOML parser; text it rejects keeps the old
quote trimming.
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* operation: probe octet-stream uploads for gzip like untyped ones
The 128-byte gzip probe ran only when the caller passed no MimeType, so an
upload labelled application/octet-stream or binary/octet-stream was never
compressed. S3 PutObject and UploadPart pass the request Content-Type (or the
octet-stream that mimeDetect fills in), so every binary S3 upload skipped
compression that the same bytes get through the filer.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* operation: check the probed upload keeps the caller's content type
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* wdclient: keep a shared volume lookup alive when its first caller leaves
Concurrent misses for the same volumes share one provider lookup, but it
ran on the first caller's context, so that caller's cancellation failed
every request waiting on it. The lookup now runs on a context no single
caller can cancel, and each caller stops waiting when its own context ends.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* wdclient: wait for the second caller to join before canceling the first
A fixed sleep did not prove the second caller had joined the shared lookup:
a late caller could hit the filled cache and pass without waiting. The test
now waits until the caller reads its own Done channel, which happens only
after DoChan registered it.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* filer: let the fake-master test helper serve any master
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* filer: gzip metadata log pieces before uploading them
Every metadata event carries the old and new entry with their whole chunk
lists, so a file grown by small fsynced appends logs the same chunk records
again and again: 2000 appends of 256 bytes through weed mount wrote 34.8 MB
of log. The uploader only compresses when the first 128 bytes compress, and
a log piece starts with fresh file ids and ETags, so it stored the log as is.
The filer now gzips each piece itself and keeps the compressed bytes when
they save at least 10%. After the same workload the volumes hold 3.05 MB
instead of 35.2 MB, file data included. Readers already honor the chunk's
IsCompressed flag on every path (filer-side decode, log file refs read
straight from volume servers), and the assign request sizes the bytes
actually stored.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* mount: keep the filer status in the flush create error
flushMetadataToFiler formatted the CreateEntry error with %v, so callers
classifying it saw only the message, which includes the file path. Wrap
it so the gRPC status the filer sent stays reachable.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* mount: retry a writeback metadata flush until the filer answers
With -writebackCache, close() returns before the chunk references reach
the filer. The deferred flush gave up after four attempts, about a minute
once each attempt's own filer retries are counted, then released the
handle: a longer filer outage dropped the file even though write() and
close() had succeeded, and its uploaded chunks became orphans.
Keep retrying for as long as the filer is unreachable, as the writeback
chunk uploads already do. A filer that answers with an error still ends
the flush after the bounded attempts, and an unlink of the draining file
still ends it on the next attempt.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* mount: retry only an unanswered filer call without a bound
The unbounded phase used util.IsTransientError, which falls back to
matching the error text. A refusal the filer sends in its response
(read-only, name too long, a plain error string) has no gRPC status, and
the flush error carries the file path, so a file named like
"unavailable-transport.log" kept retrying against a filer that answered.
Retry without a bound only when the call failed with a gRPC Unavailable
status: the dial, the connection or the mount's call deadline gave out.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* s3api: deny signed reads that name a version file by its physical path
A stored version lives at <key>.versions/v_<id>, and that path is also a
valid object key. Anonymous reads already refuse it; signed GET, HEAD,
copy sources and object subresources did not, so a caller without
s3:GetObjectVersion, or under a Deny on the logical key, could read any
version through it. Apply the same check in the shared entry recheck.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3api: recheck a version file key before answering a conditional read
A conditional GET or HEAD answered 304 or 412 with the ETag before the
handler's entry recheck, so the physical version path still confirmed a
version's ETag to signed callers.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3api: look up version file keys in the lookup-based policy recheck
Retention, legal hold, tagging writes and deletes only fetched the entry
for the recheck when the bucket had a policy, so without one they still
acted on a version file named by its physical path. Fetch it whenever
the key has the version file shape.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3api: refuse a directory-marker delete that names a version file
A trailing slash kept "<key>.versions/v_<id>/" out of the version file
shape check, and the lookup recheck misses the entry for that form, so a
DELETE reached deleteDirectoryMarker and removed a chunkless version or
delete marker without s3:DeleteObjectVersion.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
The ring tests used 127.0.0.1:18889 as an unreachable peer, but its gRPC
port 28889 is a common filer port: on a shared CI host a live filer there
accepted the forwarded exclusive create, and
TestCreateEntryExclusiveLeavesNonOwner failed. The peer now listens on a
port the test owns and drops every connection.
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* vector: share one Lance session across reads
Each request opened its table in a session of its own, so every query
loaded and decoded the index again. Get, list and query now open in one
session the server keeps, sized by -index-cache-mb and
-metadata-cache-mb; writes keep a session of their own.
Lance keys some of its cached entries, such as a version's index list,
by location and version alone, and an index dropped and created again at
its location numbers its versions from 1 again: a shared session answered
a query on the new table with the old table's index. CreateIndex now
marks each table with an id in its schema metadata, and the session
starts over when a location turns up holding another table.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* vector: share a version lease among the reads of that version
Every get, list and query tagged the version it read and deleted the tag
afterwards: two filer writes and a manifest read per request. Reads of
one version now share one tag, which goes 10 s after the last read took
it; a tag is reused for at most 5 minutes, so a busy version never
reaches the reapers' TTLs.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* vector: keep opened tables for the next read
A read built the table's store and read its latest manifest each time.
Reads now keep the opened table by location and encryption, up to 1024,
and move it to the latest version, which lists the versions and takes
the manifest from the session. A handle opened in an earlier session is
not reused.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* vector: key shared leases by manifest, and take one per version at once
A table dropped and created again within the linger reused, for a
version number the old table had, a lease whose tag the drop had
deleted. Leases are now keyed by the manifest's ETag as well. Reads that
arrive before a version has a lease wait for the first to take it,
rather than each writing a tag.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* vector: drop kept tables when the shared session starts over
A kept table holds the session it was opened in, so a replaced session
and its caches lived on in the handles of other tables until they were
evicted.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* vector: prune shared lease slots only when they double
Every lease lookup scanned all slots under the map's lock, a cost that
grew with the versions read in the last linger.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* vector: shrink the lease slot map after a prune
After a burst the map kept its peak allocation, which every later prune
scanned.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* vector: keep a lease slot a read has taken through a prune
A read that took a slot and had not yet locked it could see the slot
pruned, and then pin through it while the next read pinned through a
new slot, writing a second tag.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* admin: restyle the UI as a calm console
Grey canvas with white sheets, a docked sidebar that marks the current
page with an indigo tint, ink primary buttons, tinted status pills,
dense tables with tabular numbers and spectrum capacity meters. The
styles are design tokens layered over Bootstrap's own variables, with
light and dark values, so the page templates stay unchanged.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* admin: add a light, dark and automatic theme switch
The choice is kept in localStorage and applied before the page paints,
on the sign-in page too; automatic follows the system setting.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* admin: drop hard-coded colors from page templates
The volume id link, utilization track and plugin form borders used
fixed light-theme colors; sparklines now use the data palette.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* admin: theme the plugin job detail panel background
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* admin: keep the selected radio dot visible in dark mode
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* admin: cycle the theme from memory when storage is blocked
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* admin: keep success progress bars green
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* admin: keep keyboard focus visible on the toolbar togglers
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* admin: use contrasting text on danger and success buttons in dark mode
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* admin: lead the dashboard with cluster health and one row of tiles
The header now says in one line whether the cluster is healthy and how
many masters, volume servers and filers report. The two rows of stat
tiles become one: the volume size limit and EC volume count move into
footnotes, the mount client card joins the cluster counts, and the
counts link to their pages instead of sitting in nested cards. The
duplicate version list and the bucket shortcut in the header are gone,
and the section menus become plain View all links.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* admin: export the user columns the table actually has
The users table has username, email and access key columns, but the
export waited for six cells per row and so always produced only the
header line.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* admin: drop table titles that repeat the page title
List pages titled their only table with the page title again. The
table now sits right under the summary tiles, and the export actions
that were hidden in the table header menu move to the page header as
an Export button, matching the cluster pages. The policies menu linked
to nothing and is removed.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* admin: drop clock and duplicate summary tiles
Buckets, users, policies and service accounts spent a summary tile on
the refresh time, which the page footer already shows (policies gains
that footer). The users page also showed its user count twice.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* admin: show unset bucket settings as a dash
Rows of No owner, No quota and Not configured made the bucket table
hard to scan. Unset cells now show a quiet dash that keeps the wording
as a tooltip and for screen readers, and the name and creation date no
longer wrap (the full timestamp is in the date's tooltip).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* admin: export buckets through the page's API exporter
admin.js loads after the page, so its copy of exportBucketList replaced
the buckets page's own one. That copy scraped the visible rows, so it
exported only the current page, without CSV quoting, and picked up the
display text of the cells. The page's exporter reads every bucket from
the API with full timestamps and escaped fields.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* admin: escape quotes in the user export
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* admin: download the bucket export as a Blob
The exporter put the CSV into a data URL with encodeURI, which leaves #
alone, so a bucket owner containing # ended the download there. Use the
shared downloadCSV helper instead.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* admin: count masters as configured, not reporting
The master list comes from raft membership, which keeps a follower
that is down, so it does not show which masters are reporting.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* admin: link the Prometheus metrics endpoint from the sidebar
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* admin: drop the sidebar metrics link
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* s3api: record the caller on AssumeRole sessions
The session name is the caller's choice; the session now carries the
caller itself as the user it was issued to, as the web identity, LDAP
and credential paths already do.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3api: take a role session's aws:username from its user, not its name
An assumed-role ARN ends in the session name the caller picked, so
naming a session after another user resolved ${aws:username} and
${aws:userid} to that user in bucket and attached policies. Resolve
them from the user the session was issued to instead.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* iam: take a role session's aws:username from its user, not its name
IsActionAllowed set aws:username to the last segment of an assumed-role
ARN, the caller-chosen session name, for role and bucket policies. Use
the user the session was issued to, and drop a forwarded value when the
session names none.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3api: pass a role session caller's user on to the session it assumes
A role session that calls AssumeRole is named by its session ID, so the
new session recorded that ID as its user. An STS session caller now
passes on its own recorded user, or none.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3api: size the session claims map from the request context
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* admin: take master HTTP ports from raft member names
The masters page and dashboard turned each raft member's gRPC address
into an HTTP address by subtracting 10000 from the port, so masters with
a custom -port.grpc showed wrong or negative ports. Raft members are
named after their master address, which carries the HTTP port; use it
and keep the old derivation only for names that are not addresses.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* admin: test raft member address mapping
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* s3api: drop cached gRPC channels when a fake filer takes a port
A test that dials a dead owner leaves its failed channel in the
process-wide gRPC connection cache. When a later test's fake filer is
given the same port, its first RPC fails at once with the cached
connection refused, which is how TestRoutedMultipartFinalize failed on
the 32-bit job. The fake filer helpers now share fakeFilerAddress, which
invalidates any cached channel for the new address.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3api: test a fake filer on a dead owner's reused port
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3api: drop cached channels for dead test filer addresses too
closedFilerAddress can be handed a port whose cached channel belonged to
a fake filer that just stopped and may still look ready; then the dead
owner's transaction is sent on it instead of failing over. Build its
address with fakeFilerAddress as well.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* s3api: split upload ACL assembly out of preparePutObjectACL
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3api: store POST Object acl and grant fields on the object
The form acl and x-amz-grant-* fields were authorized but never assembled,
so the object kept no ACL: acl=private on a public-read bucket stayed
anonymously readable and invalid values were accepted. Build them with the
same rules as PutObject, which also covers the Block Public Access check.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3api: test POST Object stores its upload ACL
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3api: take the POST Object ACL from the signed form only
ACL request headers are not covered by the POST policy, so a holder of a
signed form could add one and have it stored with the upload.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3api: reject a POST acl field the policy does not cover
The acl form field now sets the stored object ACL, so like the x-amz-*
fields it needs a policy condition, as AWS requires.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3api: test unsigned ACL headers and uncovered acl fields on POST
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3api: evaluate POST upload policies on the canonical ACL
Policy conditions on the grant headers saw the raw form value, so an
equivalent spelling of a denied grantee passed. Use the PutObject request
shape that the PUT path evaluates.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3api: check bucket policy for POST ACLs with auth disabled
PutObject applies the bucket policy to an explicit upload ACL when IAM is
off; POST now does the same before storing it.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3api: reject differing acl and x-amz-acl POST fields
Both promote to the canned ACL header, so map order picked the stored ACL.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3api: test POST grant canonicalization, auth-off ACL policy, acl conflicts
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3api: check the auth-off POST ACL policy after loading the bucket
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3api: reject an empty acl or x-amz-acl POST field beside a differing one
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3api: test the auth-off POST ACL policy on a cold bucket cache and empty acl aliases
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3api: load the bucket before per-key object authorization
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3api: test the POST ACL policy on a cold bucket cache with auth on
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3api: stop per-key object authorization when the bucket fails to load
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3api: test the cold-bucket POST ACL allow path and a failed bucket load
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* lance: require UpdateTable to deregister a table
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* lance: require UpdateTable to register over an existing table
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* lance: test that GetTable alone cannot deregister or re-register a table
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* lance: a register that finds a table the caller cannot read needs UpdateTable to bring it back
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* lance: decide declare write access by whether the table existed, not by what the caller can read
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* lance: test register and declare by a caller that cannot read the table
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* lance: register clears the marker on the create branch only after checking it
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* lance: keep a register whose table was created when the marker check fails
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* filer/leveldb_handle: reopen a database whose journal latched a write error
goleveldb keeps the first journal write error and returns it for every
later write until the database is reopened, so one transient ENOSPC or
EIO fails all writes until a restart. The handle offers the leveldb.DB
methods the stores use; a write that fails on an OS error probes the
storage and, once it accepts writes and no call holds the database,
closes and reopens it and retries the write once.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* filer: open the leveldb stores through leveldb_handle
leveldb, leveldb2 and leveldb3 now recover from a transient storage
write error without a restart.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* filer: leveldb stores return iterator errors from listings and deletes
A listing on a database that cannot be read reported an empty
directory, and DeleteFolderChildren wrote an empty batch, because the
iterator error was never checked.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* filer/leveldb_handle: test a reopen that fails after the probe passes
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* filer/leveldb_handle: probe storage before draining readers
A probe that stalls in Create, Write or Sync held mu exclusively and
blocked every read on the handle. Reopen attempts are now serialized by
their own mutex, and the probe runs before the drain.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* filer/leveldb_handle: hold new calls back while a reopen drains
TryLock succeeds only at an instant with no reader, so overlapping
listings could keep a reopen from ever draining while each write spun
for the full drain timeout. The drain now holds new calls back for at
most drainTimeout; a call nested inside a running one waits that out
instead of deadlocking. A drain that times out holds off the next
attempt for busyReopenInterval, so new calls are not held back over and
over.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* filer/leveldb_handle: bound the wait for a running reopen
A write that hit the latched error waited on the reopen mutex for as long
as another caller's storage probe ran, so a stalled probe queued every
writer. Callers now wait at most reopenWait and return their error, and
still retry once a concurrent reopen succeeds.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* filer/leveldb_handle: tests wait for the drain and readers explicitly
The nested-read test polled for a drain that could start and end between
polls and then spun forever; it now syncs on the probe, widens the drain
and fails if the put returns first. The overlapping-readers test waits
until every reader holds an iterator before it writes.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* s3: authorize an empty versionId as the current-object action
GET, HEAD and DELETE treat an empty versionId as the current object, but
the action resolver keyed on the parameter's presence and authorized it
as GetObjectVersion/DeleteObjectVersion, so a Deny on GetObject or
DeleteObject did not apply.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3: test an empty versionId against a version-only policy
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3: cover a bare versionId on HEAD
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* volume: refuse a tier upload whose .dat grew during the copy
AttachRemoteDatFile only checked that the data backend was still the one
copied, so a needle write acknowledged after the copy read the .dat was
missing from the remote object, and the upload then switched reads to it
and removed the local .dat. Compare the .dat size with the copy under the
same data-file locks; on a mismatch the handler deletes the copy and keeps
the local file.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* seaweed-volume: refuse a tier upload whose .dat grew during the copy
Mirror the Go AttachRemoteDatFile check. The tier-up attach took the
write lease only inside each step, so a needle write acknowledged after
the upload read the .dat was missing from the object the volume then
switched to, and the local .dat holding it was removed. Hold one write
lease across the instance, revision and size checks, the remote-file
update, the .vif save and the backend swap, as tier-down does; a size
mismatch aborts, deleting the object and keeping the local .dat.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* volume: compare the physical .dat size before a tier attach
DiskFile pads its size past an unaligned tail, which a read-only load
keeps, while the upload copies the physical bytes. Stat the file itself
so such a volume is not refused as written during the copy.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* shell: parallel moves and a per-run limit for volumeServer.evacuate
-maxParallelization runs up to N EC shard moves at once (default 1), with the
ec.balance scheduler: never two shards of one volume at a time, free workers
go to the least busy targets. The moves are planned first with the same target
choice as before, then applied through volume_move.Mover.MoveEcShards, which
verifies the target registered a shard before deleting it on the source.
-movesPerExec limits how many volumes and EC shards one run moves (default 0
for unlimited), like -volumesPerExec in volume.balance, so a big server can be
emptied in several runs.
With -skipNonMoveable a shard that fails to move (a read error on a failing
disk, for example) no longer drops the other shards of its volume; they are
moved and the failures are listed at the end. Without it the run stops at the
first failure as before. Losing the admin lock stops the run in either mode.
Refs #12501
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* shell: plan evacuation again after failed moves; fail on a lost lock
A failed shard move kept the target slot it was planned on, so a readable
shard that found no destination in the plan stayed on the server although
the room was never used. Failed moves now give their targets back to the
model and the shards without a destination are planned again, up to
maxEcEvacuationRounds times; the failed shards are not tried again.
With -skipNonMoveable a lost admin lock was printed as a skipped failure and
the run returned success. ec.ErrLockLost is now exported and always fails
the run.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* shell: volumeServer.evacuate reports -movesPerExec after an applied EC run
The round loop only checked the limit after a failed move, so an applied run
that stopped at -movesPerExec without failures never said to run it again.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* shell: volumeServer.evacuate plans again for as long as failed moves leave room
The ten-round cap left readable shards on the server once more than ten
rounds of unreadable shards had taken the free room first, and a rerun
fails the same shards first again. Every round moves or fails each shard it
plans and a failed shard is not planned again, so the rounds end without it.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* shell: volumeServer.evacuate gives a failed move back every slot it took
When the target already held the shard in another disk type, adding it to
the planned disk type still took a slot, but the undo gave back only the
explicit one. Record the slots the plan took and restore exactly those.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* ec: an evacuation move that landed on its target keeps the target slot
A move can fail after the target registered the shard, when only the source
unmount or delete failed. Evacuate then gave the target's slot back and could
plan another shard onto a full server. The runner now checks the target after
a failed move and leaves a shard the target has out of FailedMoves.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* shell: volumeServer.evacuate gives back the slot of a failed move onto a shard the target had
A target that held the shard before the plan reports it after any failed
move, so the runner took such a move for one that landed and evacuate kept
the slot the plan had taken. The runner now lists those moves in
LandedMoves, and evacuate undoes the ones whose target had the shard, then
plans again with the room they gave back.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
* util: add WaitUntil, a cond wait bounded by a context and a timeout
The caller holds the cond lock. Cancellation and the timeout wake the
waiter through a broadcast taken under the same lock, so no wakeup is lost.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3: bound the wait for an upload slot
A PUT that arrived while the concurrent upload limits were exhausted waited
on the condition variable with no bound. The handler reads nothing from the
body while it waits, so a client that gave up and reset its connection went
unnoticed: the socket and goroutine stayed pinned until a slot freed, and a
stalled volume server grew the gateway's fd count by one per aborted PUT.
The wait now ends on InflightUploadDataTimeout with 503 SlowDown, which S3
SDKs retry, and on request cancellation. An admitted upload takes its slot
under the lock, and a finished upload broadcasts, so freed bytes admit every
waiter that fits instead of one.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* filer: bound the wait for an upload slot
The filer's PUT/POST upload limits queued requests the same unbounded way.
Reuse the bounded wait: reject with 429 after InflightUploadDataTimeout, as
the volume server does, and stop on request cancellation.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* command: add -inflightUploadDataTimeout for the S3 gateway and the filer
weed s3 -inflightUploadDataTimeout, weed filer -inflightUploadDataTimeout and
-s3.inflightUploadDataTimeout, weed server and weed mini
-filer.inflightUploadDataTimeout and -s3.inflightUploadDataTimeout. The
default of 60s matches the volume server's flag of the same name; 0 keeps
the old unbounded wait.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3: reject a queued upload without reading its body
WriteErrorResponse drains the request body before answering. Behind the
stats recorder the drain has no read deadline, and reading the body sends an
Expect: 100-continue client the go-ahead to upload what is about to be
discarded. Answer the rejection with an empty-body copy of the request so
the drain is skipped; net/http closes the connection when a large unread
body remains.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3, filer: release an upload slot under the cond lock
A waiter checks the counters under the cond lock and then waits. A release
that decremented and broadcast without the lock could land between the two,
so the waiter slept with a slot free until the next release or its timeout.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3, filer: close the HTTP/1 connection when rejecting a queued upload
net/http reads up to 256 KiB of an unread request body before sending a
keep-alive reply, so a client that stalls its body held back the 503 or
429 until the connection idle timeout. Connection: close skips that read.
HTTP/2 is left alone: there it would shut down the whole connection.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* shell: s3.clean.uploads deletes uploads over filer gRPC
Same DeleteEntry the HTTP handler runs, without minting a filer JWT,
and the gRPC path is the one a locked, routed delete needs next.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* shell: s3.clean.uploads frees an open upload under the object write lock
The cleaner deleted a stale upload recursively with no lock, so a
CompleteMultipartUpload could pass its upload-exists check after the
part entries and their chunks were gone, and publish an object over
freed chunks. Do what AbortMultipartUpload does: hold the object's
distributed write lock, and free an open upload in an ObjectTransaction
routed to the object's owner filer, conditioned on the object not
carrying the upload; if it does, delete the metadata only.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* shell: s3.clean.uploads keeps the object write lock for the whole cleanup
A large upload's recursive delete can outlast one 15 second lease, and
a completion on the distributed-lock path could then take the lock and
commit over chunks being freed. Renew the lock in the background and
skip the delete once it is lost.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* shell: s3.clean.uploads locks the path a completion commits under
For a trailing-slash key a non-versioned completion writes and locks the
entry inside the directory the key names, not the object path, so the
cleaner's transaction did not serialize with it. Lock and check the
same path the completion uses for the bucket's versioning state; the
route key stays the object's.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3api: abort locks the path a completion commits under
A routed abort locked and checked the object path, but for a
trailing-slash key a non-versioned completion commits under the entry
inside the directory the key names. Use the same path the completion
locks for the bucket's versioning state.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* shell: end s3.clean.uploads bucket failure line with a newline
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* cluster: NewBlockingLongLivedLockWithin bounds the wait on a held lock
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* shell: s3.clean.uploads skips an upload whose object lock stays held
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* cluster: share the contended lock filer setup across lock tests
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* shell: s3.clean.uploads reads the bucket's versioning state per upload
The commit path a routed completion locks depends on versioning for a
trailing-slash key. Reading it once per bucket before the run left a
window as long as the run; read it just before the delete, as the
gateway's abort does.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* mount: remove the local control socket when the mount exits
weed mount listened on /tmp/seaweedfs-mount-<hash>.sock and removed it only
when a later mount of the same directory started, so every unmount or
SIGTERM left one socket file behind. Remove it on the normal return path and
in the interrupt hook, unless a later mount of the same directory has
already replaced it.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* mount: remove the control socket before a failed mount exits
glog.Fatalf and glog.Exitf call os.Exit, which skips the deferred removal,
so a mount that failed after creating its socket (mount point not
available, fusermount failure) still left it behind.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* mount: register the socket cleanup hook before mount setup
A SIGTERM that arrived while the mount was still setting up, for example
waiting on the filer to create the mount root, exited through grace without
running the deferred cleanup. Registering the hook right after the listener
is created makes it run last, after the unmount and filesystem hooks.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* filer/etcd: scan entries from one revision
volume.fsck walked etcd one directory at a time, so a file moved from a
directory not yet listed into one already listed was seen under neither
name and its chunks were reported as orphans. The etcd store now
implements filer.EntryScanner: every page is read at the revision of the
first read, and a compacted revision fails the scan instead of falling
back to the walk.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* filer/etcd: read hard links at the scan revision
Hard-link records resolved during a scan were read at the current
revision, so a link updated meanwhile mixed two instants. Reads made with
the scan's context now use its revision.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* shell: list etcd among the stores volume.fsck reads from a snapshot
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* filer/etcd: skip hard-link records in a scan
Hard-link records live beside the entries under their random ids, so a
root scan could read one as an entry with a path made of random bytes,
which the traversal stream cannot send. Skip hard-link ids and keys whose
directory is not valid UTF-8.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* filer/etcd: recognize a hard-link record by the id it stores
A hard-link id can be valid UTF-8, so the key shape alone let some
records through a root scan as phantom files. The record's value is the
shared entry, whose HardLinkId is the record's own key; no entry key
can equal its own hard-link id.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* filer/etcd: scan entries under an invalid-UTF-8 directory
The filer sanitizes names but stores directories as given, so a path
written over HTTP can leave entries the listing never reaches. Skipping
them let volume.fsck count their chunks as orphans. Send them under the
sanitized directory instead, the substitution the filer uses for names.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* filer/etcd: fail a scan whose sanitized path names another entry
An entry under an invalid-UTF-8 directory is reported under the sanitized
directory. When that path is also another stored entry at the scan revision,
or another sanitized entry in the same scan, acting on the reported path
(volume.fsck deleting it) would reach the other file, so the scan fails and
names the stored path instead.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* ec: don't index ev.Shards[0] when the last shard was unmounted
* ec: skip a runtime emptied before the scrub merge reads its shards
FindAllEcVolumes releases each location lock before MergeEcRuntimes runs,
so an unmount can leave a merged runtime with no shards and the width
scan indexed Shards[-1].
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* ec: treat a scrub merge with no shard left as a vanished volume
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* ec: read Shards once in ShardSize so a concurrent unmount cannot empty it
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* ec: read each runtime's Shards once in the scrub merge
The vanished check and the slot construction read Shards separately, so
an unmount between them produced a non-nil view with no shards.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* ec: anchor the scrub merge on a generation that still holds shards
An unmount can empty the newest runtime while an older one keeps its
shards, which left a merged view anchored on the emptied generation with
no slots and the holder reported as skipped.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Mansur Akhmadi <10739460+mansur-akhmadi@users.noreply.github.com>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* shell: fs.mergeVolumes leaves in-progress multipart parts in place
CompleteMultipartUpload reads the part entries and later commits the
object with their fids, guarded only by the upload directory. A merge
that moved a part chunk in between deleted the source needle the new
object pointed at. Parts stay where they are until the upload completes
or is cleaned, and their chunks still count as referenced sources.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* shell: note in fs.mergeVolumes help that upload parts stay in place
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* shell: match upload parts when the buckets folder is the root
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* shell: count chunks behind a part's manifest as referenced sources
A skipped part whose chunks on a merge source sit behind a manifest was
not counted, so the merge printed no skip line and warned that nothing
references the source.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* shell: keep a part's direct source references when its manifest is unreadable
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* shell: count chunks resolved before an unreadable part manifest
ResolveChunkManifest returns the data chunks of earlier manifests along
with a later manifest's error; record them before returning so the
unreferenced-source warning does not fire for a source they reference.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* shell: test the partial manifest result without a timing delay
The unreadable manifest is now nested under a readable part manifest, so
the first level resolves fully before the failing lookup runs.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* filer_pb: add a depth-first traversal that can resume after a path
TraverseDfs walks a tree in listing order and reports each finished
path, so a caller can save one path and continue from it later. Pages
are read whole before visiting, and the entries of a page are visited
by up to five workers.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* filer.meta.backup: resume an interrupted metadata copy
The walk saves the last finished path every few seconds, and a walk
that continues an unfinished one starts after it. A fresh walk clears
the saved path first, and a finished walk clears it with the walk
start.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* filer.meta.backup: add -entriesPerSecond to throttle the metadata copy
Each entry the full copy reads waits on a rate limiter, and pages are
listed only after the previous page is copied, so the flag also bounds
how fast the copy lists the source filer. 0 keeps it unlimited.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* filer_pb: report each file of a depth-first walk once all before it are done
A throttled walk only reported progress after a whole listing page, so a
slow copy could go many minutes without a resumable path. Visit entries in
runs that end at a directory and report each file as soon as every entry
before it is done.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* volume: fail a vacuum step that finds a compaction already running
Compact, CompactByIndex and CommitCompact returned success when another
compaction held the volume. A second, uncoordinated vacuum (an expired
admin job still running, a shell volume.vacuum next to the worker) then
went on to commit or clean up the first run's .cpd/.cpx. Return a
"volume N is compacting" error instead, and refuse cleanup while a copy
is in flight, as the Rust volume server already did.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* volume: check the compaction files before a commit closes the volume
CommitCompact closed the needle map and data file and wrote the .cpc
marker before it found .cpd/.cpx missing. A duplicate commit, such as
the second of two overlapping vacuums, then left the volume registered
with no open files: it dropped out of the heartbeat and looked lost to
volume.list and volume.fix.replication until a restart. Check for the
files first so the refused commit leaves the volume as it was.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* volume: run makeupDiff before a commit closes the volume
CommitCompact closed the needle map and data file before makeupDiff, so
a makeupDiff failure (a compaction revision another commit moved, an I/O
error) returned with them closed: the volume stayed registered but
missing from the heartbeat until a restart. makeupDiff reads the live
files through its own handles while the commit holds the write lock, so
close them only once it has succeeded, as the Rust volume server does.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* volume: log a volume the heartbeat skips because its files are closed
A volume left without its needle map or data file vanished from the
heartbeat with no log line, so the master saw a replica disappear while
its files were still on disk. Warn on each skip, as the deleted-.dat
case next to it already does.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* volume: claim the compaction flag while cleanup removes the temp files
Cleanup only read the flag, so a compaction could start between the
check and the removal and lose its .cpd/.cpx. Claim it for the cleanup
instead, in Go and Rust.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* filer.sync: move the watermark over jobs that finished ahead of an older one
The watermark only took the finishing job's timestamp when it was the
oldest in flight, so jobs that completed before an older one were never
recorded and a quiet stream kept the offset behind finished events.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* filer.sync: keep one finished timestamp per gap behind a running job
A long job kept every later completion in the done heap. Only the newest
one between two running or failed timestamps can become the watermark.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* filer.sync: test done-heap compaction across several running and failed jobs
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* s3: answer GETs canceled before the first byte
A stream that fails before writing anything now always gets an S3 error
response, canceled errors included. The idle timeout and a shared lookup
canceled by another request both cancel the request context while the
client is still connected, and returning silently sent that client a
200 with Content-Length: 0. A client that has gone away never sees the
write.
Refs #12500
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3: mark inline GET write errors as written; bound the idle-cancel test
An inline object's write error comes after the status line, so the
caller must not write an S3 error on top of it. The idle-cancel test
caps the client wait and gives the response write its own deadline,
keeping only the read side on the 200 ms idle timeout.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3: test a GET whose request is canceled before the first byte
Drives GetObjectHandler through a chunk lookup that cancels the request
context, so a handler that returns before answering shows up as the empty
200 again.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3: check the cancel test reached the chunk lookup
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
* volume: forward no Content-MD5 for a JPEG the orientation fix rewrote
With -images.fix.orientation the primary stores and forwards the rotated
bytes, but sent replicas the client's digest of the original upload. The
replica checks Content-MD5 against what it receives and rejects the write,
so replicated JPEG uploads with a digest failed. The Rust volume server
already drops the digest once the body is rewritten.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* volume: test the forwarded Content-MD5 through PostHandler and a replica
The test now posts each upload to a primary whose fake master names a
real replica volume server, so it covers the JPEG gate, ReplicatedWrite
and the replica's own Content-MD5 check, and reads the stored bytes back.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* volume: fail the replica forwarding test instead of hanging when nothing is forwarded
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* volume: compare the primary and replica bytes in the replica forwarding test
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* filer.copy: compute chunk md5 so files get a real ETag
Chunks uploaded by filer.copy had no ETag, so every multi-chunk file got
the md5 of the empty string plus -N. An overwritten file kept its old ETag
and a revalidating client got 304 for changed content.
* filer.copy: store chunks in offset order
Chunks are collected in upload completion order, and the entry ETag
hashes the chunk ETags in stored order. Identical files copied with
concurrent chunk uploads got different ETags, and a file with swapped
equal-sized chunks could keep the old one.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* filer.copy: test chunk order and chunk md5 on chunked uploads
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* weed/shell: support -noLock flag in volumeServer.evacuate (#12501)
* shell: return flag parse errors from volumeServer.evacuate
A typo such as -noLock=invalid printed the parse error but the command
still succeeded, so a piped script treated the evacuation as done.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* shell: volumeServer.evacuate -h prints usage without failing
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* helm: set replication and disk on buckets created by the chart
* helm: test bucket replication and disk without ttl
* helm: reject unquoted bucket replication values
* helm: treat a null bucket replication as unset
A blank replication key in a values file failed the render with the
quoted-string error, while blank ttl and disk are skipped.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* helm: require a three-digit bucket replication
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* helm: note that removing bucket settings keeps the stored rule
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* helm: reject bucket replication above 255
fs.configure rejects an xyz value above 255, so the hook Job failed after
the release rendered.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* fix(shell): fs.log.purge purges every filer, not just one
Each filer keeps its own metadata logs in its own store, so purging a
single filer leaves the chunks referenced by the others and the volume
vacuum cannot reclaim anything.
The shell picked that single filer at random (commands.go:208) when no
-filer flag was given, and the admin script worker pinned it to
filers[0] (admin_script_handler.go:574), so the same command purged a
different filer on every run and never the whole cluster.
fs.log.purge now resolves the filer list from the master and purges each
of them, continuing when one is unreachable and reporting the failures
at the end. -filer=<addr>,... restricts the purge to a subset.
Also stops mutating the shared command environment per filer, so a
long-lived shell keeps its original filer.
* fix(shell): bound filer discovery and cover the cluster path
Address review feedback:
- resolveLogPurgeFilers waited on the master with context.Background(): a
shell whose master connection dropped would hang before purging
anything, where it used to purge the connected filer. Bound the master
round trip with a timeout and fall back to the connected filer when the
master cannot be reached or lists no filer.
- the tests only exercised the single-filer fallback. Rename that test to
what it actually covers and add one asserting every filer of the list is
attempted and the shared command environment keeps its original filer.
* shell: reject an fs.log.purge -filer list with no addresses
A comma or whitespace only -filer value fell through to master
discovery and purged every filer instead of the intended subset.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* shell: fall back to the shell's filer without a second master lookup
When the master listed no filer, FilerAddress() ran filer discovery again
and could wait out the shell's ConnectTimeout before failing. Also cover
master discovery with the fake discovery master.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* shell: test the fs.log.purge filer loop against a fake filer
The previous test looped over the addresses itself, so it passed no
matter what Do did.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Martin Morez <martin@morez.org>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* filer: validate bucket names before the SQL DDL hooks
The SQL store builds CREATE TABLE and DROP TABLE statements by formatting
the bucket name into the query. The DML paths already reject names that
fail VerifyS3BucketName, but the bucket-creation and bucket-deletion
hooks ran the DDL without that check, so a name reaching them through the
filer or WebDAV directory path (which skips the check for dot-prefixed
directories) could inject stacked statements. Apply the same validation
the DML paths use.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* filer: check that a valid bucket runs both DDL statements
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
* weed-db: end startup when password authentication fails
The failure branches returned the result of writing the error response,
which is nil on success, so the handshake carried on to ReadyForQuery.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* weed-db: test startup with wrong and unknown credentials
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* weed-db: check the auth error code and fail on stray read errors in the startup test
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* admin: escape file properties before rendering the modal
The Properties modal concatenated the entry name, path, MIME type and
extended attributes into HTML, so object metadata written over S3 could
inject markup into the admin page.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* admin: keep every property key when escaping
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* security: do not accept a path- or method-scoped filer token as admin
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* security: test that scoped filer tokens fail the admin token check
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* s3tables: let a table renamed across namespaces keep committing
* s3tables: keep the same-directory exception off a directory another table shares
A record written before metadata locations were confined to their namespace
(6f93f874) and claimed (#12473) can keep its files in another table's
directory, in another namespace. The exception that lets a table renamed
across namespaces keep committing to the directory it left behind accepted
such a record's commits too, where master refused them.
The namespace check now runs first, as on master. Only a location it refuses
is matched against the record's current directory, and the exception holds
only when no other record keeps its files there. Where other records' entries
sit is ignored, so a table created under the renamed table's old name, whose
files go to a directory of its own, does not take the exception away. The
scan runs only on that path, so commits inside the namespace cost what they
did.
Co-Authored-By: Claude <noreply@anthropic.com>
* s3tables: seed the table at the renamed table's old name
A rename may keep the old name taken, so creating a table there can be
refused; seeding the record keeps the test on the update path.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude <noreply@anthropic.com>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
filer: prefix bucket default collection, assign copy collection explicitly
S3 server-side copy paths (CopyObject/UploadPartCopy) assign volumes with an empty collection and relied on the filer's bucket fallback. The fallback used the raw bucket name, while PutObject uses getCollectionName(bucket) = <filerGroup>_<bucket>, so copies landed in a second, unprefixed collection.
Prefix the filer's bucket default with the filer group, and have the S3 copy path pass that collection explicitly, like PutObject. Direct filer writes keep path-rule precedence, so this only aligns S3 copy with S3 put.
Adds regression tests for the fallback, explicit collection precedence, and the S3 copy collection resolver.
* weed/s3api: avoid truncated marker rescan in ListObjects (#12540)
Signed-off-by: Tyagiquamar <mohdquamartyagi@gmail.com>
* fix(s3): validate marker directory candidates
Respect prefix filters and internal-directory exclusions when recovering marker candidates. Restrict lookups to direct children, propagate read failures other than not-found, and deduplicate directory marker echoes.
Add regression coverage and model promoted objects consistently in the XML-unsafe prefix fixture.
---------
Signed-off-by: Tyagiquamar <mohdquamartyagi@gmail.com>
Co-authored-by: Chris Lu <chris.lu@gmail.com>
* s3vectors: bound body buffering before authentication
* docs: include vector engine installation and source builds
* s3vectors: prevent MIME sniffing of JSON error responses
* docs: load vector Cargo configuration when building from source
* docs: note the native protoc requirement for Windows vector builds
* docs: make vector source checkout and working directory explicit
* ci: handle deb822 sources during Ubuntu mirror failover
* ci: bound Ubuntu mirror attempts and reject partial index updates
* vector: expose request metrics and registration-aware health endpoints
* vector: report immutable read cache and storage fetch metrics
* docs: describe vector engine health, metrics, and maintenance
* vector: count canceled requests when their handler is dropped
* vector: account for uncached large immutable storage reads
* vector: reliably release the stalled credential reader in tests
* vector: observe large reads as their streamed body is consumed
* vector: bound heartbeats on established master registration streams
* test: exercise hardened vector maintenance on Go and Rust volumes
* vector: register engines in their configured filer group
* vector: make retried index creation preserve compatible datasets
* vector: allow registration to advertise a separate address
* test: release stalled credential readers without blocking FIFO cleanup
* test: allow hardened vector coverage in a named filer group
* vector: discover engines in the same filer group as their gateway
* vector: fall back when the advertised address is empty
* admin: expose vector readiness, maintenance, and management tools (#12520)
* s3vectors: share vector discovery with trusted management clients
* admin: show vector engine readiness and recent maintenance activity
* admin: suggest S3 Vectors IAM actions from the supported operations
* admin: manage vector indexes, policies, tags, encryption, and query data
* admin: validate SSE-S3 support before creating encrypted vector buckets
* admin: discover vector engines in their registration group
* admin: probe vector engines concurrently with independent time budgets
* admin: verify encryption capability on the remote S3 gateway
* admin: defer discovery until after the HTTP server can start
* admin: distinguish unknown vector health services from failed engines
* admin: count workers that can execute vector index maintenance
* admin: allow read-only users to query vectors and inspect tags
* vector: scope gateway and admin discovery to the configured filer group
* admin: refresh policy state after mutations and serialize policy controls
* vector: drain active RPCs before closing replaced discovery connections
* vector: respect request cancellation while waiting for discovery
* vector: prefer another registered engine after a transport failure
* admin: check all masters before reporting no registered vector engines
* iam: honor attached vector policies without broadening legacy grants
* vector: allow short-lived discovery clients to close their connections
* vector: refresh cached master addresses for each discovery lookup
* iam: enforce vector bucket tag conditions in attached policies
* iam: authorize vector bucket creation with the requested tags
* test: exercise vector IAM grants and tag denies on Go and Rust volumes
* admin: reject authenticated vector mutations without an admin role
* test: run vector IAM assertions before the shared engine is stopped
* vector: add native mini startup, shell tools, and a runnable Compose example (#12522)
* s3vectors: share vector discovery with trusted management clients
* admin: show vector engine readiness and recent maintenance activity
* admin: suggest S3 Vectors IAM actions from the supported operations
* admin: manage vector indexes, policies, tags, encryption, and query data
* admin: validate SSE-S3 support before creating encrypted vector buckets
* admin: discover vector engines in their registration group
* admin: probe vector engines concurrently with independent time budgets
* admin: verify encryption capability on the remote S3 gateway
* admin: defer discovery until after the HTTP server can start
* admin: distinguish unknown vector health services from failed engines
* admin: count workers that can execute vector index maintenance
* admin: allow read-only users to query vectors and inspect tags
* vector: scope gateway and admin discovery to the configured filer group
* admin: refresh policy state after mutations and serialize policy controls
* vector: drain active RPCs before closing replaced discovery connections
* vector: respect request cancellation while waiting for discovery
* vector: prefer another registered engine after a transport failure
* admin: check all masters before reporting no registered vector engines
* iam: honor attached vector policies without broadening legacy grants
* vector: allow short-lived discovery clients to close their connections
* vector: refresh cached master addresses for each discovery lookup
* iam: enforce vector bucket tag conditions in attached policies
* iam: authorize vector bucket creation with the requested tags
* test: exercise vector IAM grants and tag denies on Go and Rust volumes
* admin: reject authenticated vector mutations without an admin role
* test: run vector IAM assertions before the shared engine is stopped
* mini: optionally start and supervise the vector engine
* mini: bootstrap vector buckets with existing Lance catalog helpers
* shell: manage vector buckets, indexes, and vector data
* docs: provide a runnable vector and maintenance worker Compose stack
* shell: scope vector discovery to the session and close its connections
* mini: honor the bind address for vector and metrics listeners
* mini: reserve vector ports before assigning gRPC and compatibility listeners
* mini: wait for vector storage setup and master registration
* mini: create requested vector buckets without requiring the Lance HTTP listener
* shell: refresh master addresses during vector discovery
* mini: resolve sibling vector executables with platform extensions
* vector: preserve uncertain creates and fix mini listener allocation (#12525)
* vector: retain catalog records when index creation has an unknown outcome
* mini: keep chosen listener ports reserved throughout allocation
* mini: resolve hostname binds before starting vector listeners
* vector: fix integration test imports and recover interrupted index creation (#12533)
* shell: register vector operations from the command package
* s3vectors: resume interrupted index initialization safely
* mini: exclude disabled WebDAV from listener allocation
* s3vectors: retain uncertain creates across refused retries
* s3vectors: claim pending indexes before retrying initialization
* s3vectors: compare resolved distance metrics during recovery
* s3vectors: preserve metadata keys when retrying initialization
* s3vectors: verify ownership before initializing new catalog records
* helm: deploy vector engines with TLS, probes, and monitoring
* helm: provision vector buckets through the S3 Vectors API
* helm: bracket IPv6 pod addresses for vector listeners
* helm: discover vector engines through the all-in-one master
* helm: accept vector component NetworkPolicy overrides
* helm: bind vector metrics to the pod address on both IP families
* helm: provision vector buckets through embedded and all-in-one gateways
* docs: use the full Helm name for external vector certificates
* helm: render vector instance labels as strings
* helm: allow vector bucket hooks to use restricted security contexts
* helm: honor vector security context enable switches
* helm: prevent duplicate vector monitoring labels
* helm: reject metrics flags with the default pre-metrics vector image
* test: skip vector chart compatibility checks when Helm is unavailable
* test: pass installer scripts through stdin on Windows
* ci: allow cold Intel macOS vector builds to finish
* ci: retain completed Rust builds when later platform checks fail
* ci: build, test, and publish vector engines for macOS and Windows
* installer: install native vector archives on macOS and Windows
* ci: run desktop vector checks for shared Rust and protobuf changes
* ci: publish vector releases from a separate job with write permissions
* ci: build tagged vector releases without restoring caches
* ci: propagate Windows bash path conversion failures
* grafana: add vector API, engine, cache, and maintenance panels
* helm: include vector panels with Kubernetes metric selectors
* grafana: preserve cluster and namespace identity when matching engine health
* grafana: retain engine identity when scrapes fail
* grafana: show zero cache hits before the first hit
* grafana: keep cluster and namespace identity in vector cache ratios
* topology: list collections without nesting the collection map read lock
Items() already takes the read lock and copies the values. Holding it
around the call made ListCollections take the read lock twice, so a
DeleteCollection waiting for the write lock in between blocked the
second read lock and wedged the master.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* topology: test ListCollections against a concurrent collection delete
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-08 15:05:41 -08:00
875 changed files with 74298 additions and 7916 deletions
@@ -17,7 +17,9 @@ SeaweedFS is a simple and highly scalable distributed file system. There are two
1. to store billions of files!
2. to serve the files fast!
One `weed` binary serves an S3 object store, a POSIX file system, and a lakehouse with S3 Tables, all over the same data. Each blob is one disk read away, capacity grows by starting another volume server, and cloud storage can be cached or tiered transparently. Both read and write operations have O(1) complexity and can run at the full speed supported by the underlying hardware.
One SeaweedFS cluster serves an S3 object store, a POSIX file system, a lakehouse with S3 Tables and a built-in Iceberg REST Catalog, and [vector search with S3 Vector Buckets](#vector-search-with-s3-vector-buckets), all on shared storage. Each blob is one disk read away, capacity grows by starting another volume server, and cloud storage can be cached or tiered transparently. Blob reads and writes have O(1) complexity and can run at the full speed supported by the underlying hardware.

- [Download Binaries for different platforms](https://github.com/seaweedfs/seaweedfs/releases/latest)
For SeaweedFS Enterprise binaries, pass `--enterprise` to the script: `... | bash -s -- --enterprise`.
Then start a ready-to-use S3 object store:
```bash
@@ -204,12 +209,13 @@ On one laptop, [`weed benchmark`][Benchmarks] writes 1KB files at 15,700 per sec
## The most complete S3 API ##
The S3 gateway implements the object, bucket, S3 Tables, IAM, and STS APIs on one endpoint, so the AWS SDKs and CLI, rclone, restic, Spark, and Trino work unchanged.
The S3 gateway implements the object, bucket, S3 Tables, S3 Vectors, IAM, and STS APIs on one endpoint, so the AWS SDKs and CLI, rclone, restic, Spark, and Trino work unchanged. S3 Vectors uses the `weed-vector` service described [below](#vector-search-with-s3-vector-buckets).
| API | Operations |
| --- | --- |
| S3 bucket and object | 73 |
| S3 Tables | 36 |
| S3 Vectors | 19 |
| IAM | 39 |
| STS | 5 |
@@ -223,7 +229,7 @@ The full operation list is in [Amazon S3 API][AmazonS3API], and [Supported APIs
## A data warehouse with S3 Tables ##
SeaweedFS is a lakehouse in one system. [S3 Table Buckets][S3TableBucket] hold Apache Iceberg tables by default, or [Lance][LanceCatalog] tables for vectors and multimodal data, and the built-in [Iceberg REST Catalog][IcebergCatalog] and Lance namespace serve them directly. There is no Hive Metastore, Glue, or separate catalog service to deploy, secure, and back up.
SeaweedFS is a lakehouse in one system. [S3 Table Buckets][S3TableBucket] hold Apache Iceberg tables by default, or [Lance][LanceCatalog] tables for vectors and multimodal data, and the built-in [Iceberg REST Catalog][IcebergCatalog] and Lance namespace serve them directly. The S3 Tables API and Iceberg REST Catalog share the same namespaces, table metadata, and permissions. There is no Hive Metastore, Glue, or separate catalog service to deploy, secure, and back up.
* Query engines operate on the same tables at the same time: [Spark][SparkIceberg], [Trino][TrinoIceberg], [Dremio][DremioIceberg], [DuckDB][DuckDBIceberg], [Apache Doris][DorisIceberg], [RisingWave][RisingWaveIceberg], ClickHouse, and [LanceDB][LanceDB]. Catalog commits are atomic compare-and-swap, so concurrent writers are safe. [Lakekeeper][Lakekeeper] can front the same storage with STS-vended credentials. [OLake Go][OLake] is a high-performance, open-source EL (Extract–Load) platform that bridges operational databases and open lakehouse storage - writing Apache Iceberg tables or Parquet files on Amazon S3.
* [Automated table maintenance][IcebergMaintenance]: compaction, snapshot expiration, orphan file removal, and manifest rewriting, configured per bucket or table through the S3 Tables maintenance APIs, and the same for [Lance][LanceMaintenance].
@@ -232,6 +238,40 @@ SeaweedFS is a lakehouse in one system. [S3 Table Buckets][S3TableBucket] hold A
`S3_TABLE_BUCKET=warehouse ./weed mini -dir=./data` brings the whole stack up on a laptop.
## Vector search with S3 Vector Buckets ##
Keep embeddings alongside your objects and tables in the same SeaweedFS cluster. S3 Vector Buckets expose the AWS S3 Vectors API for semantic search, retrieval-augmented generation (RAG), and recommendations, using Lance tables on SeaweedFS storage.
* Create vector buckets and indexes; insert, retrieve, list, and delete vectors with the AWS SDKs or CLI pointed at your S3 endpoint.
* Find the nearest vectors with cosine or Euclidean distance, filter by metadata, and return distances and metadata with the results.
* Control access with `s3vectors:*` IAM actions and vector bucket policies; tag buckets and indexes for organization and access control.
Install the separate vector engine; the default installer installs only `weed`:
Release 4.53 provides vector binaries for Linux amd64 and arm64. If your release has no binary for your platform, build from [source](seaweed-vector). From a directory where you want a new checkout:
If you already have a checkout, start at its root and skip the clone and `cd seaweedfs` steps.
For Windows source builds, set `PROTOC` to a native `protoc.exe` before running Cargo; the [desktop build workflow](.github/workflows/rust-vector-platforms.yml) shows how to select the vendored executable.
Run the installed `weed-vector` binary alongside the local cluster started with `weed mini` above (or use `seaweed-vector/target/release/weed-vector` after building from source):
```bash
weed-vector --masters=localhost:9333
```
It registers with the master and the S3 gateway discovers it automatically. An AWS S3 Vectors client uses the same S3 endpoint at http://localhost:8333. The vector engine stores its data through the cluster's filers and volume servers, so embeddings share the storage layer's replication and erasure coding. See the [AWS SDK examples](test/s3vectors/s3vectors_test.py) for bucket, index, and query operations.
## A fast cache for cloud storage ##
[Cloud Drive][CloudDrive] mounts a bucket from S3, Google Cloud Storage, Azure, Backblaze B2, Wasabi, Storj, or any S3-compatible store into SeaweedFS and serves it at local speed:
With `rules` left empty the route sends all traffic to the component's own Service and port. A rule may set `matches`, `filters`, `timeouts` and `backendRefs`, and a rule without `backendRefs` still routes to that Service. In all-in-one mode the master, volume, filer and s3 routes target the all-in-one Service. When S3 runs only on the filer (`filer.s3.enabled` without `s3.enabled`), the s3 route uses `filer.s3.port`.
Outside all-in-one mode, the volume Ingress and the default volume route target the `client-volume` Service, which lists only Ready volume pods. The `volume` Service also publishes pods that are not Ready yet, so their DNS records can exist before readiness. An explicit `backendRefs` entry that names the `volume` Service can therefore reach a volume server before it is Ready. Name the `client-volume` Service instead, with `volume.port`, and copy its `metadata.name` from `helm template` because long release names are truncated.
## Network Policies
In a namespace with a default-deny policy the install hangs: the components cannot resolve each other, and the post-install bucket hook waits on the master and filer until it gives up. `networkPolicy.enabled` renders one `NetworkPolicy` per component, selecting its pods by the standard `app.kubernetes.io/{name,instance,component}` labels and admitting traffic from the other pods of the release on the ports that component listens on.
@@ -610,6 +612,17 @@ Two things worth knowing before you turn this on:
The DNS selectors default to CoreDNS as kubeadm, kind and the managed offerings from AWS, Google and Azure install it. On OpenShift, override `egress.dnsNamespaceSelector` and `egress.dnsPodSelector` to match `openshift-dns`; see the comment in `values.yaml`.
## Volume server HTTPS
With `global.seaweedfs.enableSecurity`, gRPC between components uses mutual TLS, but the volume servers' HTTP port, which carries the file data, stays plaintext. Set `global.seaweedfs.volumeHttps.enabled: true` to serve it over TLS:
- `[https.volume]` points at the volume certificate, which then also names every volume group's service (`*.<fullname>-volume.<namespace>`), since volume servers register as `<pod>.<fullname>-volume.<namespace>`. With `certificates.externalCertificates`, the provided `<fullname>-volume-cert` needs those names.
- `[https.client]` is enabled with the client certificate and the chart CA, so every chart pod that talks to the volume servers does it over HTTPS.
- The volume probes use HTTPS.
- Every pod that reads `security.toml` gets a `checksum/volume-https` annotation, so switching the option on or off restarts them together.
The volume servers do not require client certificates, because kubelet probes cannot present one; enable JWT signing (`securityConfig.jwtSigning`) to authorize volume reads and writes. `[https.client]` also sends the admin UI's HTTP lookups on masters and S3 over HTTPS, which their plain listeners refuse, so the UI can miss volume public URLs and refuse SSE-S3 vector buckets. A volume Ingress or HTTPRoute needs its controller's HTTPS-backend setting. `weed mount`, the CSI driver and other clients outside the chart need their own `[https.client]` with the chart CA. The option is not supported with `allInOne`, whose volume server registers by pod IP.
## Common labels
`commonLabels` adds labels to the metadata of every object the chart creates. It may be a map or a template string rendering to one. The chart's own labels cannot be overridden, and pods (see `podLabels`) and `volumeClaimTemplates` are left unchanged.
@@ -687,3 +700,47 @@ generates a new UUID and the license stops matching. See
[Master data](#master-data-hostpath-vs-a-claim). Check the binding with
The S3 gateway serves the S3 Vectors API on its existing endpoint. Enable a vector engine and Lance maintenance worker alongside the cluster:
```yaml
s3:
enabled: true
vector:
enabled: true
replicas: 1
worker:
enabled: true
admin:
enabled: true
secret:
adminPassword: change-me
global:
seaweedfs:
securityConfig:
jwtSigning:
filerWrite: true
```
The engine registers its pod address with the masters and stores indexes through the filers and volume servers. Both Go and Rust volume servers use the same setup. The maintenance worker builds and updates ANN indexes; queries can scan data before indexing completes. Enabling filer write signing also requires clients writing directly to the filer HTTP API to supply a signed JWT.
The bundled engine images support Linux amd64 and arm64. Use `vector.imageOverride` to select a compatible engine build, `vector.masters` for an external cluster, and `vector.resources` to set memory and CPU limits. The cache defaults to 512 MiB and is configured with `vector.readCacheMB`.
`global.seaweedfs.enableSecurity` provisions and mounts the vector certificate and shared security configuration. With external certificates, provide the `<fullname>-vector-cert` secret (`my-seaweedfs-vector-cert` for release `my`). `fullname` follows `seaweedfs.fullname`, including `fullnameOverride`, `nameOverride`, and DNS-name truncation. Provide this secret alongside the existing CA and client secrets.
The default `vector.metricsPort: 0` keeps TCP probes. Set `vector.metricsPort: 9329` to serve `/metrics`, `/health`, and `/ready` from the engine; readiness then checks master registration. Images older than 4.54 do not accept the metrics flag, so leave metrics off when pinning `image.tag` or `vector.imageOverride` to one. Enabling `global.seaweedfs.monitoring.enabled` also creates the vector ServiceMonitor. NetworkPolicy includes the configured gRPC and metrics ports.
To provision vector buckets during installation and upgrades, set:
```yaml
vector:
enabled: true
createBuckets:
- embeddings
```
The hook uses the AWS CLI's S3 Vectors API and works with existing SeaweedFS vector releases. It creates missing buckets, preserves existing vector buckets, and fails if the name belongs to an Iceberg bucket. It uses the chart's generated S3 admin credentials, including `s3.credentials.admin.existingSecret`. With a custom `s3.existingConfigSecret` or an external endpoint, set `vector.bucketHook.credentialsSecret` to a Secret containing `AWS_ACCESS_KEY_ID` and `AWS_SECRET_ACCESS_KEY` (and optionally `AWS_SESSION_TOKEN`). Key names are configurable. `vector.bucketHook.endpoint` overrides the internal S3 service URL; `caBundleSecret` and `caBundleKey` configure a private HTTPS CA. The hook image and resource limits are configurable under `vector.bucketHook`. For restricted Pod Security, select a non-root-compatible image and configure `vector.bucketHook.podSecurityContext` and `containerSecurityContext`, including `runAsNonRoot` and the image user ID.
"description":"Vector engine metrics require HTTP metrics to be enabled. Gateway bucket labels require vector bucket attribution.",
"fieldConfig":{
"defaults":{
"color":{
"mode":"palette-classic"
},
"custom":{
"axisBorderShow":false,
"axisCenteredZero":false,
"axisColorMode":"text",
"axisLabel":"",
"axisPlacement":"auto",
"barAlignment":0,
"drawStyle":"line",
"fillOpacity":0,
"gradientMode":"none",
"hideFrom":{
"legend":false,
"tooltip":false,
"viz":false
},
"insertNulls":false,
"lineInterpolation":"linear",
"lineWidth":1,
"pointSize":4,
"scaleDistribution":{
"type":"linear"
},
"showPoints":"auto",
"spanNulls":false,
"stacking":{
"group":"A",
"mode":"none"
},
"thresholdsStyle":{
"mode":"off"
}
},
"mappings":[],
"thresholds":{
"mode":"absolute",
"steps":[
{
"color":"green",
"value":null
}
]
},
"unit":"reqps",
"unitScale":true
},
"overrides":[]
},
"gridPos":{
"h":8,
"w":12,
"x":0,
"y":140
},
"id":96,
"links":[],
"options":{
"legend":{
"calcs":[],
"displayMode":"list",
"placement":"bottom",
"showLegend":true
},
"tooltip":{
"mode":"single",
"sort":"none"
}
},
"pluginVersion":"8.1.2",
"targets":[
{
"datasource":{
"type":"prometheus",
"uid":"${DS_PROMETHEUS}"
},
"editorMode":"code",
"expr":"sum by (type, bucket) (rate(SeaweedFS_s3_request_total{namespace=~\"$NAMESPACE\",type=~\"s3vectors\\\\..*\"}[$__rate_interval]))",
"range":true,
"refId":"A",
"legendFormat":"{{type}} {{bucket}}"
}
],
"title":"Vector API requests by bucket",
"type":"timeseries"
},
{
"datasource":{
"type":"prometheus",
"uid":"${DS_PROMETHEUS}"
},
"description":"Vector engine metrics require HTTP metrics to be enabled. Gateway bucket labels require vector bucket attribution.",
"fieldConfig":{
"defaults":{
"color":{
"mode":"palette-classic"
},
"custom":{
"axisBorderShow":false,
"axisCenteredZero":false,
"axisColorMode":"text",
"axisLabel":"",
"axisPlacement":"auto",
"barAlignment":0,
"drawStyle":"line",
"fillOpacity":0,
"gradientMode":"none",
"hideFrom":{
"legend":false,
"tooltip":false,
"viz":false
},
"insertNulls":false,
"lineInterpolation":"linear",
"lineWidth":1,
"pointSize":4,
"scaleDistribution":{
"type":"linear"
},
"showPoints":"auto",
"spanNulls":false,
"stacking":{
"group":"A",
"mode":"none"
},
"thresholdsStyle":{
"mode":"off"
}
},
"mappings":[],
"thresholds":{
"mode":"absolute",
"steps":[
{
"color":"green",
"value":null
}
]
},
"unit":"s",
"unitScale":true
},
"overrides":[]
},
"gridPos":{
"h":8,
"w":12,
"x":12,
"y":140
},
"id":97,
"links":[],
"options":{
"legend":{
"calcs":[],
"displayMode":"list",
"placement":"bottom",
"showLegend":true
},
"tooltip":{
"mode":"single",
"sort":"none"
}
},
"pluginVersion":"8.1.2",
"targets":[
{
"datasource":{
"type":"prometheus",
"uid":"${DS_PROMETHEUS}"
},
"editorMode":"code",
"expr":"histogram_quantile(0.95, sum by (le, bucket) (rate(SeaweedFS_s3_request_seconds_bucket{namespace=~\"$NAMESPACE\",type=\"s3vectors.QueryVectors\"}[$__rate_interval])))",
"range":true,
"refId":"A",
"legendFormat":"{{bucket}}"
}
],
"title":"Vector query p95 latency",
"type":"timeseries"
},
{
"datasource":{
"type":"prometheus",
"uid":"${DS_PROMETHEUS}"
},
"description":"Vector engine metrics require HTTP metrics to be enabled. Gateway bucket labels require vector bucket attribution.",
"fieldConfig":{
"defaults":{
"color":{
"mode":"palette-classic"
},
"custom":{
"axisBorderShow":false,
"axisCenteredZero":false,
"axisColorMode":"text",
"axisLabel":"",
"axisPlacement":"auto",
"barAlignment":0,
"drawStyle":"line",
"fillOpacity":0,
"gradientMode":"none",
"hideFrom":{
"legend":false,
"tooltip":false,
"viz":false
},
"insertNulls":false,
"lineInterpolation":"linear",
"lineWidth":1,
"pointSize":4,
"scaleDistribution":{
"type":"linear"
},
"showPoints":"auto",
"spanNulls":false,
"stacking":{
"group":"A",
"mode":"none"
},
"thresholdsStyle":{
"mode":"off"
}
},
"mappings":[],
"thresholds":{
"mode":"absolute",
"steps":[
{
"color":"green",
"value":null
}
]
},
"unit":"reqps",
"unitScale":true
},
"overrides":[]
},
"gridPos":{
"h":8,
"w":12,
"x":0,
"y":148
},
"id":98,
"links":[],
"options":{
"legend":{
"calcs":[],
"displayMode":"list",
"placement":"bottom",
"showLegend":true
},
"tooltip":{
"mode":"single",
"sort":"none"
}
},
"pluginVersion":"8.1.2",
"targets":[
{
"datasource":{
"type":"prometheus",
"uid":"${DS_PROMETHEUS}"
},
"editorMode":"code",
"expr":"sum by (type, code) (rate(SeaweedFS_s3_request_total{namespace=~\"$NAMESPACE\",type=~\"s3vectors\\\\..*\",code=~\"[45]..\"}[$__rate_interval]))",
"range":true,
"refId":"A",
"legendFormat":"{{type}} {{code}}"
}
],
"title":"Vector API errors",
"type":"timeseries"
},
{
"datasource":{
"type":"prometheus",
"uid":"${DS_PROMETHEUS}"
},
"description":"Vector engine metrics require HTTP metrics to be enabled. Gateway bucket labels require vector bucket attribution.",
"fieldConfig":{
"defaults":{
"color":{
"mode":"palette-classic"
},
"custom":{
"axisBorderShow":false,
"axisCenteredZero":false,
"axisColorMode":"text",
"axisLabel":"",
"axisPlacement":"auto",
"barAlignment":0,
"drawStyle":"line",
"fillOpacity":0,
"gradientMode":"none",
"hideFrom":{
"legend":false,
"tooltip":false,
"viz":false
},
"insertNulls":false,
"lineInterpolation":"linear",
"lineWidth":1,
"pointSize":4,
"scaleDistribution":{
"type":"linear"
},
"showPoints":"auto",
"spanNulls":false,
"stacking":{
"group":"A",
"mode":"none"
},
"thresholdsStyle":{
"mode":"off"
}
},
"mappings":[],
"thresholds":{
"mode":"absolute",
"steps":[
{
"color":"green",
"value":null
}
]
},
"unit":"short",
"unitScale":true
},
"overrides":[]
},
"gridPos":{
"h":8,
"w":12,
"x":12,
"y":148
},
"id":99,
"links":[],
"options":{
"legend":{
"calcs":[],
"displayMode":"list",
"placement":"bottom",
"showLegend":true
},
"tooltip":{
"mode":"single",
"sort":"none"
}
},
"pluginVersion":"8.1.2",
"targets":[
{
"datasource":{
"type":"prometheus",
"uid":"${DS_PROMETHEUS}"
},
"editorMode":"code",
"expr":"last_over_time(SeaweedFS_vector_connected{namespace=~\"$NAMESPACE\"}[5m]) * on (cluster, namespace, instance, job) up{namespace=~\"$NAMESPACE\"}",
"range":true,
"refId":"A",
"legendFormat":"{{instance}}"
}
],
"title":"Registered vector engines",
"type":"timeseries"
},
{
"datasource":{
"type":"prometheus",
"uid":"${DS_PROMETHEUS}"
},
"description":"Vector engine metrics require HTTP metrics to be enabled. Gateway bucket labels require vector bucket attribution.",
"fieldConfig":{
"defaults":{
"color":{
"mode":"palette-classic"
},
"custom":{
"axisBorderShow":false,
"axisCenteredZero":false,
"axisColorMode":"text",
"axisLabel":"",
"axisPlacement":"auto",
"barAlignment":0,
"drawStyle":"line",
"fillOpacity":0,
"gradientMode":"none",
"hideFrom":{
"legend":false,
"tooltip":false,
"viz":false
},
"insertNulls":false,
"lineInterpolation":"linear",
"lineWidth":1,
"pointSize":4,
"scaleDistribution":{
"type":"linear"
},
"showPoints":"auto",
"spanNulls":false,
"stacking":{
"group":"A",
"mode":"none"
},
"thresholdsStyle":{
"mode":"off"
}
},
"mappings":[],
"thresholds":{
"mode":"absolute",
"steps":[
{
"color":"green",
"value":null
}
]
},
"unit":"reqps",
"unitScale":true
},
"overrides":[]
},
"gridPos":{
"h":8,
"w":12,
"x":0,
"y":156
},
"id":100,
"links":[],
"options":{
"legend":{
"calcs":[],
"displayMode":"list",
"placement":"bottom",
"showLegend":true
},
"tooltip":{
"mode":"single",
"sort":"none"
}
},
"pluginVersion":"8.1.2",
"targets":[
{
"datasource":{
"type":"prometheus",
"uid":"${DS_PROMETHEUS}"
},
"editorMode":"code",
"expr":"sum by (type, code) (rate(SeaweedFS_vector_request_total{namespace=~\"$NAMESPACE\",code!=\"OK\"}[$__rate_interval]))",
"range":true,
"refId":"A",
"legendFormat":"{{type}} {{code}}"
}
],
"title":"Vector engine request failures",
"type":"timeseries"
},
{
"datasource":{
"type":"prometheus",
"uid":"${DS_PROMETHEUS}"
},
"description":"Vector engine metrics require HTTP metrics to be enabled. Gateway bucket labels require vector bucket attribution.",
"fieldConfig":{
"defaults":{
"color":{
"mode":"palette-classic"
},
"custom":{
"axisBorderShow":false,
"axisCenteredZero":false,
"axisColorMode":"text",
"axisLabel":"",
"axisPlacement":"auto",
"barAlignment":0,
"drawStyle":"line",
"fillOpacity":0,
"gradientMode":"none",
"hideFrom":{
"legend":false,
"tooltip":false,
"viz":false
},
"insertNulls":false,
"lineInterpolation":"linear",
"lineWidth":1,
"pointSize":4,
"scaleDistribution":{
"type":"linear"
},
"showPoints":"auto",
"spanNulls":false,
"stacking":{
"group":"A",
"mode":"none"
},
"thresholdsStyle":{
"mode":"off"
}
},
"mappings":[],
"thresholds":{
"mode":"absolute",
"steps":[
{
"color":"green",
"value":null
}
]
},
"unit":"short",
"unitScale":true
},
"overrides":[]
},
"gridPos":{
"h":8,
"w":12,
"x":12,
"y":156
},
"id":101,
"links":[],
"options":{
"legend":{
"calcs":[],
"displayMode":"list",
"placement":"bottom",
"showLegend":true
},
"tooltip":{
"mode":"single",
"sort":"none"
}
},
"pluginVersion":"8.1.2",
"targets":[
{
"datasource":{
"type":"prometheus",
"uid":"${DS_PROMETHEUS}"
},
"editorMode":"code",
"expr":"sum by (type) (SeaweedFS_vector_in_flight_requests{namespace=~\"$NAMESPACE\"})",
"range":true,
"refId":"A",
"legendFormat":"{{type}}"
}
],
"title":"Vector engine requests in flight",
"type":"timeseries"
},
{
"datasource":{
"type":"prometheus",
"uid":"${DS_PROMETHEUS}"
},
"description":"Vector engine metrics require HTTP metrics to be enabled. Gateway bucket labels require vector bucket attribution.",
"fieldConfig":{
"defaults":{
"color":{
"mode":"palette-classic"
},
"custom":{
"axisBorderShow":false,
"axisCenteredZero":false,
"axisColorMode":"text",
"axisLabel":"",
"axisPlacement":"auto",
"barAlignment":0,
"drawStyle":"line",
"fillOpacity":0,
"gradientMode":"none",
"hideFrom":{
"legend":false,
"tooltip":false,
"viz":false
},
"insertNulls":false,
"lineInterpolation":"linear",
"lineWidth":1,
"pointSize":4,
"scaleDistribution":{
"type":"linear"
},
"showPoints":"auto",
"spanNulls":false,
"stacking":{
"group":"A",
"mode":"none"
},
"thresholdsStyle":{
"mode":"off"
}
},
"mappings":[],
"thresholds":{
"mode":"absolute",
"steps":[
{
"color":"green",
"value":null
}
]
},
"unit":"percentunit",
"unitScale":true
},
"overrides":[]
},
"gridPos":{
"h":8,
"w":12,
"x":0,
"y":164
},
"id":102,
"links":[],
"options":{
"legend":{
"calcs":[],
"displayMode":"list",
"placement":"bottom",
"showLegend":true
},
"tooltip":{
"mode":"single",
"sort":"none"
}
},
"pluginVersion":"8.1.2",
"targets":[
{
"datasource":{
"type":"prometheus",
"uid":"${DS_PROMETHEUS}"
},
"editorMode":"code",
"expr":"(sum by (cluster, namespace, job, instance) (rate(SeaweedFS_vector_cache_requests_total{namespace=~\"$NAMESPACE\",result=\"hit\"}[$__rate_interval])) or on (cluster, namespace, job, instance) (0 * sum by (cluster, namespace, job, instance) (rate(SeaweedFS_vector_cache_requests_total{namespace=~\"$NAMESPACE\"}[$__rate_interval])))) / sum by (cluster, namespace, job, instance) (rate(SeaweedFS_vector_cache_requests_total{namespace=~\"$NAMESPACE\"}[$__rate_interval]))",
{{- fail "global.seaweedfs.volumeHttps is not supported with allInOne: its volume server registers by pod IP, which the volume certificate cannot name" }}
{{- $preStopSeconds := .Values.s3.preStopSeconds | int }}
{{- $minGracePeriod := add $preStopSeconds 15 }}
{{- if and (gt $preStopSeconds 0) (lt (.Values.s3.terminationGracePeriodSeconds | int) $minGracePeriod) }}
{{- fail (printf "s3.terminationGracePeriodSeconds (%v) must be at least %d: the gateway keeps serving for s3.preStopSeconds (%d) after SIGTERM, then gives in-flight requests up to 15s to finish; add the time any s3.lifecycle.preStop hook takes" .Values.s3.terminationGracePeriodSeconds $minGracePeriod $preStopSeconds) -}}
{{- end }}
apiVersion:apps/v1
kind:Deployment
metadata:
@@ -49,6 +54,7 @@ spec:
{{- end }}
annotations:
{{- toYaml $podAnnotations | nindent 8 }}
{{- with include "seaweedfs.volumeHttpsAnnotation" $ }}{{ . | nindent 8 }}{{ end }}
{{- if and $volume.enabled (eq (include "seaweedfs.componentName" (list $ $volumeName)) $name) }}
{{- fail (printf "The volume Ingress/HTTPRoute Service %q has the same name as the %s Service after truncation. Shorten fullnameOverride or the release name." $name $volumeName) }}
<titleid="title">SeaweedFS: one cluster for objects, files, tables, and vectors</title>
<descid="description">S3 object storage, a POSIX file system, table buckets with a built-in Iceberg REST Catalog and Lance namespace, and S3 Vector Buckets share one SeaweedFS storage cluster. Masters coordinate the cluster, filers manage namespaces and metadata, and volume servers store replicated or erasure-coded blobs. Cloud caching, cloud tiering, and cross-cluster replication extend the cluster.</desc>
@@ -45,6 +45,8 @@ Rule of thumb: start with 3 masters, then grow volume servers with data.
- Metadata store options: PostgreSQL, MySQL, Redis, Cassandra, TiKV, etc.
- Run **3 filers** for HA
- Filers are stateless relative to each other — state lives in the DB
- No quorum among filers: 2 already survive one failure; the third keeps a spare while one is down for upgrades
- Give clients every filer address (e.g. `-filer=f1:8888,f2:8888,f3:8888`) or a load balancer that forwards both the HTTP and gRPC ports (8888 and 18888)
**S3 Servers**
@@ -77,7 +79,7 @@ Keeps housekeeping off the hot path.
`weed-vector` serves the S3 Vectors data plane through the S3 gateway. Indexes use Lance tables on the cluster's filers and volume servers. Go and Rust volume servers use the same engine and API.
Point an AWS S3 Vectors client at the S3 endpoint, normally `http://localhost:8333`. The engine registers with the masters automatically. `--listen` defaults to `127.0.0.1:8090`; use an address reachable from the gateway when running on another host or in a container.
The HTTP listener provides `/health`, `/ready`, and `/metrics`. Readiness requires an active registration with the master. The gRPC listener also implements the standard health service for `vector_pb.VectorService` and the whole server. These health statuses follow master registration.
Metrics use the `SeaweedFS_vector_` prefix:
-`connected`: active master registration.
-`request_total{type,code}`, `request_seconds{type}`, and `in_flight_requests{type}`: engine requests and their gRPC outcomes.
-`query_results_total`: vectors returned by queries.
-`cache_requests_total{result}`, `cache_bytes`, and `cache_entries`: immutable read cache activity and occupancy.
-`storage_read_bytes_total` and `storage_read_seconds`: storage fetches for cacheable immutable reads.
The S3 gateway separately publishes `SeaweedFS_s3_` request metrics with `type="s3vectors.<Operation>"`. Scrape both processes to distinguish gateway latency from engine work. Metrics and HTTP probes are unauthenticated; bind them to an internal address or restrict their network access.
For mutual TLS, configure `[grpc.vector]` for the engine's server certificate, `[grpc.client]` for its master/filer connections, and the cluster CA. HTTPS volume reads also use `[https.client]`. The engine follows the same signing and encryption configuration as the other cluster clients.
Run the Lance maintenance worker against the same masters and Lance namespace to build and update ANN indexes. Queries can scan data that has not been indexed yet. See the [maintenance worker documentation](../seaweed-worker/README.md) and [Helm chart](../k8s/charts/seaweedfs/README.md) for deployment settings.
The integration suite at `test/s3vectors` runs the same API and observability checks against both Go and Rust volume servers, including mutual TLS, signed volume reads, and encrypted volume data.
r#"json_extract(`metadata`, '$["k"]') = '"v"' AND json_exists(`metadata`, '$["k"]') AND json_get_string(`metadata`, 'k') = 'v' AND json_array_contains(`metadata`, '$["k"]', '"v"') AND NOT json_array_contains(`metadata`, '$["k"]', key)"#,
"{} AND TRY_CAST(json_extract(`metadata`, '$[\"n\"]') AS DOUBLE) > 3 AND NOT json_exists(`metadata`, '$[\"a.b\\\"c\"]') AND COALESCE(json_extract(`metadata`, '$[\"f\"]') = 'true', false)",
"(COALESCE(json_extract(`metadata`, '$[\"author\"]') in ('\"alice\"', '\"bob\"', '\"x\"'), false) OR COALESCE(json_array_contains(`metadata`, '$[\"author\"]', '\"alice\"'), false) OR COALESCE(json_array_contains(`metadata`, '$[\"author\"]', '\"bob\"'), false) OR COALESCE(json_array_contains(`metadata`, '$[\"author\"]', '\"x\"'), false))",
Path::Ivf,
),
(
"{\"author\":{\"$in\":[\"alice\",7]}}",
"(COALESCE(json_extract(`metadata`, '$[\"author\"]') in ('\"alice\"'), false) OR COALESCE(TRY_CAST(json_extract(`metadata`, '$[\"author\"]') AS DOUBLE) in (7), false) OR COALESCE(json_array_contains(`metadata`, '$[\"author\"]', '\"alice\"'), false) OR COALESCE(json_array_contains(`metadata`, '$[\"author\"]', '7'), false))",
Loaded 100 of 875 files, more files were not shown because too many files have changed in this diff.
Show more
Reference in new issue
Block a user
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.