* shell: fix s3.circuitBreaker help examples and error messages
The help examples used -type count, but -type accepts Count and MB (as
its flag description says), so both examples failed with "type not
valid, only 'count' and 'bytes' are allowed", an error that names
neither accepted value. Two other errors pointed at flags that do not
exist: -max and -counts, where the flag is -values.
Use -type Count in the examples, name Count and MB in the type error,
and -values in the value errors. TestCircuitBreakerHelpExamplesRun runs
every help example and fails on master for the two -type count lines.
Signed-off-by: anant <170962632+antcybersec@users.noreply.github.com>
* shell: name the bad action and the -actions flag in s3.circuitBreaker
The unknown-action error pointed at a nonexistent -action flag and echoed
the whole -actions list instead of the action that was rejected.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Signed-off-by: anant <170962632+antcybersec@users.noreply.github.com>
Co-authored-by: anant <170962632+antcybersec@users.noreply.github.com>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* filer: report a SQL listing that fails partway through
ListDirectoryPrefixedEntries never checked rows.Err() after its rows.Next()
loop. When the result set failed partway, for example on a lost
connection, rows.Next() returned false and the listing returned the
entries read so far with a nil error, so callers saw a shorter directory
instead of an error. Check rows.Err() once the loop has run to the end,
as the scan in abstract_sql_store_scan.go already does. A listing the
callback stopped on purpose is left alone: rows.Err() would return a
canceled request context's error there.
This covers every store built on AbstractSqlStore (mysql, mysql2,
postgres, postgres2, sqlite).
Signed-off-by: anant <170962632+antcybersec@users.noreply.github.com>
* filer: wait for the canceled SQL listing to release its rows in the test
A fixed sleep let the callback return before database/sql closed the
rows, so a listing that always consulted rows.Err could still pass.
Wait, bounded, until the connection is back in the pool instead.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Signed-off-by: anant <170962632+antcybersec@users.noreply.github.com>
Co-authored-by: anant <170962632+antcybersec@users.noreply.github.com>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
The test unmarshalled the command's output into both actual and expect,
so it compared the output with itself and the expected JSON in each case
was never checked; it also skipped the first case. Compare every case's
output with its expected config as S3CircuitBreakerConfig messages (the
output includes default fields such as "enabled": false that the
expected JSON leaves out), and restore LoadConfig when the test ends.
Signed-off-by: anant <170962632+antcybersec@users.noreply.github.com>
Co-authored-by: anant <170962632+antcybersec@users.noreply.github.com>
* dailyrun: count entries read from the filer, skipped ones included
A versioned bucket's pre-pass and a resumed walk read entries without
emitting them, so a walk can read for hours with no visited entry.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3 lifecycle: count listed entries as progress work
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
A flush uploads the dirty pages without the handle lock, then commits the
entry under it. A write in between grows the entry's size, but its pages
are not in the committed chunks. Clearing dirtyMetadata then made the next
flush skip the commit, leaving a file with the new size and no chunk for
the write, which reads back as zeros.
Count page writes on the handle and clear dirtyMetadata only when none
landed since the flush started uploading.
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* vacuum: report a request dropped because another vacuum is running
The master runs one vacuum at a time. A request that arrives while
another one is running, the automatic vacuum or an earlier request, is
dropped: Topology.Vacuum logs "Vacuum is already running" and returns.
The VacuumVolume RPC and /vol/vacuum then answered as if the vacuum had
run, so volume.vacuum printed nothing and exited 0 with no volume
compacted. Where the automatic vacuum is busy for hours, an explicit
volume.vacuum -volumeId could do nothing every time, and nothing said so.
Topology.Vacuum now returns ErrVacuumRunning for a dropped request. The
RPC returns it and /vol/vacuum answers 409 with it. volume.vacuum names
the request that failed, because a list of volumes or collections is
sent as several requests and a vacuum that starts in between fails only
the later ones. Requests are still dropped, not queued, and the
automatic vacuum ignores the error as before.
The integration tests that need their /vol/vacuum request to run ask
again while the master answers 409.
Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
* docs: note the 409 answer from /vol/vacuum while another vacuum runs
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
* s3 lifecycle: report work-based job progress during a pass
The daily pass sent progress only while starting, so a healthy scan that
ran for many hours could reach the admin's 24-hour no-progress guard.
Worker heartbeats prove the worker is alive, not that this job moves.
While dailyrun.Run is in progress the handler now sends a progress update
every 5 minutes, and only when the pass's counters (entries visited,
replay events, RPC outcomes including errors) moved since the last
delivered update. An idle tick sends nothing, so a stalled pass still
expires. A failed send is retried on the next tick. The reporter is
stopped and joined before the completion is sent, so no progress can
follow the terminal result.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* admin: test that progress, not heartbeats, keeps an old job active
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3 lifecycle: label progress phase and bucket as the last started
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* s3 lifecycle: add a LifecycleReady RPC that only authorizes the caller
LifecycleReady runs the same checkAdminAuth as LifecycleDelete and reads
no bucket or object metadata, so a lifecycle worker can confirm that an S3
front accepts its internal credentials before it starts a pass.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3 lifecycle: check S3 fronts with LifecycleReady before the pass
Execute used to dial the first registered S3 front and start loading
policies and walking buckets, so a missing or mismatched
jwt.filer_signing.key surfaced only as repeated per-bucket delete
failures. It now checks each front in order with LifecycleReady, using
the credentials LifecycleDelete sends, under a 10s bound that also
follows parent cancellation. The first front that passes serves the
pass; fronts that fail are reported in the job's progress and the log.
When none passes, the job fails before any listing with an error that
names the front and says whether this worker has the key set, without
key or token material. Fronts that predate LifecycleReady are checked
with an empty LifecycleDelete, which they authorize and then reject
before any lookup.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* s3 lifecycle: collect per-pass bucket results and work counters
WalkBuckets kept only the first bucket error and logged the rest. A
RunStats shared by the walker, the walker dispatcher and the replay
drain now records every bucket's walk result (walked, failed, skipped
on cancellation), entry and object visits, replay events, and
LifecycleDelete outcomes under the dispatch_total outcome labels.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3 lifecycle: persist the last fully successful pass
Shard cursors and walk timestamps advance on partial passes as well, so
neither says when lifecycle last covered every policy. Keep that in its
own record next to the cursors, plus a worker gauge for it.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3 lifecycle: report each pass's result in the job record
The job ended with a fixed success summary, or with only the first error,
and malformed lifecycle policies were V(1) logs. The completion now
carries the pass status (complete, partial, failed, canceled), bucket
totals and failures, invalid policies, entry, object and event counts,
LifecycleDelete outcomes, the last phase and bucket, the elapsed time
and the last full pass, on failed passes as well.
Only a complete pass advances the full-pass record; a malformed policy
makes the pass partial.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* admin: keep a failed job's result in its record
Completions with Success=false dropped Result.Summary and
Result.OutputValues, so a handler could not report what a failed run
had done, which buckets failed, or how far it got. Keep them; the error
message still drives the job's message and run record.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* grafana: chart the time since the last full lifecycle pass
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3 lifecycle: a replay halted on an unresolved delete is a partial pass
BLOCKED, RETRY_LATER and exhausted transport retries stop a shard's
replay without an error, leaving its due events for a later pass. Count
those shards and report the pass as partial, so it no longer advances
the full-pass record.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3 lifecycle: a pass with throttled walks is not a full pass
WalkerInterval can skip the walk for walk-only rules (ExpirationDate,
ExpiredDeleteMarker, NewerNoncurrent) with no error. Count those shards
and keep the full-pass record for passes that evaluated every rule; the
job reports full_pass and walks_deferred.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3 lifecycle: keep the saved full-pass time when the save fails
Publishing a timestamp that never reached the filer let the gauge move
back on the next job's reload. Keep the previous value and report the
save error in the job result.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* admin: drop a failed attempt's summary when the retry reports none
Retries reuse the job record, and failed completions now keep their
result summary, so a retry that completes without a Result showed the
failed attempt's summary, and a successful one used it as its message.
Clear the summary when a completion carries no Result, as the output
values already are.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* mount: fence a dropped handle only when it holds unpublished content
dirtyMetadata is also set by chmod, xattr and fallocate, so a failed
metadata-only close refused every later write of the inode.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* mount: fence the inode before its last handle leaves the handle map
A reopen between the map removal and the fence could publish at the
stale kernel EOF.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* mount: fence a failed metadata flush only when the handle holds content
A refused synchronous flush or an abandoned writeback metadata flush of a
handle with only chmod, xattr or fallocate changes fenced the inode, so
later writes were refused until FORGET.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* mount: count a size change on an open handle as unpublished content
A truncate or extend through an open handle changes the entry's size and
chunks, but it set only dirtyMetadata. A failed close then skipped the
final-release fence, so a reopen before FORGET could publish against the
untruncated entry.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* mount: mark copied content dirty before each page of a streaming copy
A copy that failed or was cancelled after earlier pages were accepted left
dirtyContent false, so the final-release fence skipped that content.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* mount: mark copied content dirty only after a page is accepted
A copy whose first page is rejected leaves the destination unchanged and
the kernel EOF where it was, so fencing the inode on release only blocked
later writes until FORGET.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* mount: skip the server-side copy while the destination holds unpublished content
A whole-file server copy clears dirtyContent when it commits, so a page a
cancelled streaming copy left behind would lose its release fence. Fall
back to the streaming copy instead.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* mount: skip the server-side copy while the source holds unpublished content
The filer copies the stored source, so pages still pending on the source
handle would be missing from the destination. The streaming copy reads
them through the dirty pages.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* mount: check the publication fence before a server-side whole-file copy
The filer copy publishes the destination without going through the
flush-time guard, so a fenced writeback inode could still take new
content before FORGET. Fall back to the streaming copy, whose flush is
guarded.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
filer: take the listing snapshot as a claim below later and unfinished stamps
A ListEntries snapshot tells the client that every event at or below it is
already in the listing, so the mount drops those events from a directory
build and fences them with the directory floor. The snapshot was a raw
time.Now(), which an event stamp can tie on a coarse clock: on Windows a
mkdir right after a listing of its parent was stamped with the snapshot's
own timestamp, the mount took it as listed, and the new directory stayed
missing from a parent marked cached. A create stamped before the snapshot
but not yet in the store was dropped the same way.
Take the snapshot through the same in-flight claim the idle heartbeat uses:
it stays below every unfinished stamp, and every later stamp lands above it.
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* mount: guard append gaps over concurrent content merges
* mount: fence a writeback inode when its async publication is abandoned
A drain that gives up on a non-precondition error left the token clear, so a
reopened writeback handle could publish past the stored EOF.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* mount: fence a writeback inode when its async data flush fails
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* mount: fence the inode when its last handle drops uncommitted content
A synchronous release whose final flush fails destroys the handle's dirty
pages, while under writeback the kernel EOF already counts them.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* mount: start the close()/fsync commit deadline after the data upload
Flush, Fsync and the truncate path of SetAttr create the 30 s
metadataFlushTimeout context on entry and pass it to doFlush, so the
clock also runs through FlushData, the upload of the dirty pages. That
upload has its own timeouts and retries. When an assign through a stalled
filer or a slow volume server takes most of the 30 s, CreateEntry then
fails at once with "context deadline exceeded" and close() returns EIO
for data that was already stored. The deadline is meant for the commit
alone (#10102: "a healthy CreateEntry completes in well under a second").
doFlush now takes the commit timeout and starts the deadline once the
data is uploaded. close() still cannot hang on a stuck filer: the commit
keeps its 30 s bound. Release and the flush before a rename pass 0, as
before (no overall bound, only the per-call filer timeouts).
Seen in production on a training cluster: a 30 s stall of one replica's
disk, and a filer that stalled for about 30 s, each failed close() with
EIO although the upload had succeeded.
Tests: close(), fsync() and a truncate below dirty data, each after an
assign slower than the deadline (all three fail before this change), and
a filer that never answers CreateEntry still fails close() at the
deadline.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* mount: charge the flush commit deadline only for its filer calls
Starting the 30 s deadline after the data upload is not enough.
flushMetadataToFiler does more than call the filer before it commits: it
waits for the handle's locks, reads the file back and uploads it again
when its chunks are bloated (maybeMergeChunks), and uploads a manifest
once enough chunks have built up (MaybeManifestize). None of that takes
the context. When it outlasts the deadline, CreateEntry again fails at
once with "context deadline exceeded", and close() or fsync() returns an
error for data that was stored.
The deadline becomes a budget, commitBudget, charged only while a filer
call of the commit runs: CreateEntry, and the read of a changed hard
link before it. The attempts of one flush share it, and a flush is not
retried once it is spent, so a stuck filer still fails close() after
30 s. Release, the flush before a rename and the async flush pass no
budget, as before.
Seen in production: the merge of a 1.8 MiB sqlite file with 183 chunks
took 32 s because one chunk's read first tried a volume server that was
down, and the commit failed 32 microseconds after the merge finished.
Tests: a flush whose manifest upload is slower than the deadline commits
(it fails without this change), and the filer calls of one flush share
the budget.
Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
* mount: give the stalled-lookup refusal test the new doFlush argument
TestContentFlushRefusalWithStalledLookup came in from master after this
branch changed doFlush's signature, so the merged package did not build.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
The read of the stored entry ran in the create's filer callback, so a read
that hit the call deadline turned the refusal into a filer timeout, and
both retry layers resent the refused create. Read it in its own call,
bounded to one filer call.
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* util/http: stop a full-chunk read at the chunk's recorded size
weed mount shrinks a chunk's Size in place on ftruncate but leaves its
needle untouched. A view covering the whole shortened chunk is fetched
without a Range header, so the reader returned every stored byte. The
filer GET then wrote past the declared Content-Length and sent an empty
body, and the stream reader served the stale tail.
ReadUrlAsStream and the cipher path now deliver at most size bytes;
size 0 still reads the whole needle for manifest and log chunks.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3api: limit a plain chunk read to the requested size
A full-chunk read sends no Range header, so a chunk shortened by a mount
truncate streamed its whole needle into the S3 response.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* util/http: still report a failure past the chunk size
The volume server aborts a full-needle response when the needle fails
its checksum, and only at the end. Stopping at the chunk size missed
that, so a damaged copy read as healthy. Read the rest of the body and
return its error so the caller tries another copy.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* s3vectors: call the registered weed-vector processes servers
The gateway's discovery named them workers, the name of the admin
plugin workers. Also drops two comments left without a function.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* test/s3vectors: build the Enterprise engine as weed-vector
In an Enterprise checkout, make ./weed-vector builds the distributed
engine, which also serves VectorService, instead of a separate
weed-vector-worker. An open-source checkout builds weed-vector-server
as before.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* test/s3vectors: distributed mode starts two more weed-vector engines
It started weed-vector-worker, which no target builds now, and waited
for a gateway log line that no longer exists. The Enterprise
weed-vector's engines all join the placement controller.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* mount: record a hard link's looked-up names on its inode
Lookup mapped a second name of a hard-linked file to the existing inode
without adding it to the inode's names, so GetAllPaths missed it and the
final forget left it mapped to a released inode.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* mount: keep an open file publishing when a peer unlinks one of its names
A delete event from another client marked the inode's open handle deleted
even when other names still held the hard-linked file, so every later
fsync reported success without saving metadata. The delete event's entry
carries the link count the filer decided on, so an alias delete now drops
only that name, keeps the handle on a name this mount knows, and has the
next flush read the survivor's stored entry before publishing.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* mount: test peer unlinks of a hard link name held open
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* mount: leave an unlinked file out of the periodic metadata flush
flushFileMetadata did not check the handle's delete mark. A file
unlinked while open before its deferred create reached the filer was
created again by the next periodic flush, which expects no entry.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* mount: read a pending link change before a periodic metadata flush
Only flushMetadataToFiler applied linkRefresh. After a peer unlinked one
name, the periodic flush kept publishing over the old link count, so the
filer refused every one until a regular flush ran.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* mount: test the periodic flush after a peer unlinks a hard link name
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
On Windows Lance commits local tables by rename and leaves the manifest
ETag empty, so a table dropped and written again at its location reused
the old table's lease for the same version number.
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
A peer's chunk append now merges into the flush, so these tests no longer
saw the refusal they exercise. A replaced file is still refused.
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* s3: re-check a versioned DeleteObject If-Match under the object lock
A versioned or suspended DeleteObject without a versionId routed to the
owner filer after checking If-Match outside any lock, so concurrent
conditional deletes could all pass the check and each add a delete
marker. Keep conditional deletes on the lock path, which re-checks the
precondition under the object write lock, as conditional versioned PUTs
already do.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3: guard a conditional version delete on the owner filer
DeleteObject with a versionId checked If-Match before routing the
delete to the owner filer, so concurrent conditional deletes of one
version could all report success. Carry the If-Match clause into the
owner's transaction as a guard on the version (the null entry for
versionId=null), evaluated under the object's path lock before anything
applies, and map that guard failing to 412 instead of the object-lock
403. An If-Match the filer cannot evaluate stays on the lock path.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3: carry an If-Match ETag list into the routed delete
A DeleteObject whose If-Match listed several ETags fell back to the
gateway's object write lock, while a single-ETag delete of the same key
ran under the owner filer's path lock. The two locks do not exclude each
other, so a list delete and a single-ETag delete could both report
success. Send every candidate to the filer, whose strong comparison
matches checkDeleteIfMatch, so all conditional deletes with an owner are
decided under the owner's lock.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3: fail a conditional null delete on an owner transport error
The owner may have committed the delete before the error, and the gateway
lock path it fell back to does not exclude routed deletes, so a second
If-Match delete could pass there too. Answer 500 as the specific-version
path does; unconditional deletes still fall back.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3: fail a conditional null delete on any owner error but the nested refusal
Only the owner's non-empty-folder refusal is known to apply nothing. After
any other owner error the delete may have been partly applied, and a lock-path
re-check does not exclude a concurrent routed delete of the same version, so a
conditional null-version delete answers 500 instead of falling back.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3: strip a nested conditional null delete on the owner
A conditional versionId=null delete that the owner refuses because other
keys are nested under the entry used to fall back to the gateway lock and
re-check If-Match there. That lock does not exclude routed deletes: once
the last nested key is removed, a concurrent routed delete can pass the
same check on the owner, and the fallback then treats the missing entry as
deleted, so both requests answer 204.
Demote the entry on the owner instead, in a transaction guarded by the
If-Match clauses and by the entry it was read as, with the object-lock
guards on the request. A missing or already demoted entry answers 412.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* filer: add an IF_OBJECT_EXISTS write condition
IF_EXISTS also holds for a plain directory, which is what is left of a key
once the object on it is stripped while other keys stay nested under it.
IF_OBJECT_EXISTS holds only for an entry that carries an object.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3: require an object for a conditional null delete with If-Match: *
Stripping the null version off an entry other keys are nested under leaves a
plain directory. Once the last nested key is gone, a routed delete with
If-Match: * still found an entry there and removed the empty directory, so a
second conditional delete could also answer 204. The owner now checks that
the entry carries an object.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3: settle a wildcard null delete on the entry when the filer predates IF_OBJECT_EXISTS
A filer older than IF_OBJECT_EXISTS fails it closed, so with the gateway
upgraded first every If-Match: * delete of a null version answered 412. When
the narrowed guard fails, the gateway now reads the entry on the owner: absent
or a plain directory is a 412, otherwise the file is deleted, or the directory
stripped, under IF_EXISTS and IF_ENTRY_EQUAL on the entry it read. The demote
keeps the caller's If-Match clauses, since the entry guard already requires an
object.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* filer: cancel a chunk download that no reader waits for
* filer: stop waiting for reader cache budget once cancelled
* filer: keep a cancelled download's re-lookups off its context
The refresh and retry lookups took the download's context. FilerClient
coalesces concurrent lookups of a volume, so a cancelled download leading
one failed the other readers waiting on it. Fetches still stop on cancel.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* filer: test that cancelling a download leaves its re-lookup running
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* filer: cancel unheld downloads when the reader cache is destroyed
ReaderCache.destroy waited for every downloader, so closing a reader
waited for a stalled prefetch nobody read. Downloads with no reader or
stream on them are now cancelled first; ones still in use keep running.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* filer: test that destroying the reader cache cancels an unread prefetch
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* filer: cancel a destroyed download once its last holder leaves
destroy skipped downloads a reader or stream still held, and once they
were out of the map nothing cancelled them, so destroy waited for a fetch
nobody used. removeConsumedLocked now cancels any consumable download,
whether or not it is still in the map.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* filer: test a destroyed download cancelled after its last stream leaves
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* filer: release the stream only after destroy dropped the download
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* filer: test a cancelled stream leaving a waiting reader's download running
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* s3tables: re-anchor legacy owners for federated STS sessions
A web-identity role session was keyed on its bare OIDC subject until
owners became issuer-scoped. Its sessions now carry the scoped aws:userid
but no asserted issuers, so the legacy subject check never applied and the
session lost the table buckets and namespaces it had created. Read the
configured issuer set for such sessions.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3api: keep token claims from asserting trusted issuers on STS sessions
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* iceberg: give the catalog's table handler the local accounts and issuer set
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3tables: test a federated session on a bucket its legacy subject owns
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* sts: drop a token-supplied aws:userid when a session has no parent user
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3api: take a session's aws:userid from its signed parent user
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3api: test that session claims keep only the server-asserted aws:userid
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3tables: match legacy subject patterns in table bucket policy principals
A policy written before issuer scoping can name role sessions by a pattern
over their bare OIDC subjects. Match such a pattern against the caller's
subject claim once that subject resolves to the session's scoped principal
under the single configured issuer.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3tables: test a federated session against a legacy subject pattern principal
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* s3api: refuse a PUT to a stored version's physical path
In a Suspended bucket a PUT to <key>.versions/v_<id> replaced the stored
version's body in place. Look the entry up when the key has the version
file shape and deny the write when it is a version file.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3api: refuse copy, multipart, POST and rename writes to a version file
Apply the same version file check to the CopyObject, CreateMultipartUpload,
CompleteMultipartUpload, POST Object and RenameObject destinations.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3api: read a version file write target on its logical owner
The guard resolved the key as an object, which on a ring reads the
physical path's owner and, in a versioned bucket, may return a nested
version under <key>.versions/v_<id>.versions instead of the stored
version. Look up the physical entry directly, routed by the logical key.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3api: confirm a version file miss on a skipped owner
When the owner was recently unreachable the routed lookup answers from a
replica, whose miss does not show the version is absent. Ask the owner
directly in that case and refuse the write when it cannot answer.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3api: confirm every routed version file miss on the owner
Whether the routed lookup skipped the owner can change between the lookup
and a second unreachable check, letting a replica's miss through. Ask the
owner after any miss instead.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* fix(mount): retain refused async metadata before releasing handle
* mount: sync the failed-metadata directories with util.FsyncDir
Directory fsync fails on Windows, where the mount also builds, and returns
EINVAL on some filesystems. Either kept the retention loop retrying a record
that could never be saved, so the drain never finished.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* mount: keep failed-metadata records in a per-user directory
The write cache root defaults to the shared temp directory. The first mount
to save a record created failed-metadata there as 0700, so another user's
mount could never save one and its drain retried forever.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* mount: drop the async conflict probe test
TestContentAsyncConflictKeepsRecoveryRecord drives the same refused drain
with the retry backoff stubbed out; the probe slept through the real backoff
(7 s) and its kept-handle branch could not run. Its peer-entry check moves
into the recovery record test.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* filer/etcd: client keepalive and per-request deadlines
An etcd member that goes silent with its connections open (a dead or hung
node, a network partition, a frozen process) keeps the store's requests
balanced onto it waiting forever, because the client has no keepalive and
the calls have no deadline. Writes through the filer stop although the
other members keep quorum.
keepalive_timeout (default 3s, "0s" turns keepalive off) enables gRPC
keepalive with the ping interval fixed at gRPC's 10s minimum. gRPC also
sets TCP_USER_TIMEOUT to it on Linux, so a connection whose sent data goes
unacknowledged that long is closed by the kernel. request_timeout (default
3s) bounds every Get, Put and Delete unless the caller set a deadline;
timeout keeps its meaning as the dial timeout. DeleteFolderChildren gets no
request deadline: its DeleteRange grows with the directory, and keepalive
still ends it if the member goes silent.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* Update weed/filer/etcd/etcd_store.go
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
* Update weed/filer/etcd/etcd_store_test.go
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
* Update weed/command/scaffold/filer.toml
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
* filer/etcd: no request deadline on entry writes
A Put whose client deadline expires may still have been committed. The
filer's upload handler treats the error as a failed write and deletes the
uploaded chunks, which the saved entry may then point at. InsertEntry and
UpdateEntry therefore keep the caller's context: etcd's own request
timeout ends a Put a live member cannot commit, and keepalive ends one on
a member that goes silent. Reads and deletes keep request_timeout; a
timed-out store delete returns before the filer deletes any chunks.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* filer/etcd: no request deadline on KvPut
The filer wrapper stores a hard-linked file's attributes and chunk list
through KvPut. An HTTP append to a hard-linked file whose KvPut times out
after etcd saved it returns an error, and the upload handler deletes the
appended chunks the saved hard link now points at. KvPut keeps the
caller's context like InsertEntry and UpdateEntry.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
When a mount is rooted at an existing directory, two code paths replace
that directory's filer entry with one the mount builds itself:
- every mount start re-creates the root with a non-exclusive mkdir,
which the filer handles as an update of the whole entry;
- creating or removing an entry directly in the root saves the root's
entry to touch its mtime, and maybeLoadEntry synthesizes that entry
from the mount options instead of reading the filer.
The filer keeps only crtime and inode on such an update, so the root
loses its mode, owner and extended attributes. When the root is a
bucket it also loses the bucket quota, which silently drops to 0
(unlimited), and the settings S3 keeps on the bucket entry. The explicit
allowEmptyFolders=false that #11370 meant to keep is replaced by true at
every mount start, because the mkdir has wiped it before
ensureBucketAllowEmptyFolders looks.
Create the root only when a lookup says it is missing, and then
exclusively. Leave the root out of the filer-side mtime touch: its
attributes come from the mount options (setRootAttr), so the filer copy
of its mtime was never visible through the mount anyway.
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* mount: keep flushing an open file whose other hard link survives an unlink
Unlink marks the inode's open handle deleted so no flush recreates the
removed name. When the entry was one of several hard links the inode lives
on under another name, yet the mark stayed and every later fsync and close
returned OK without publishing metadata. Lift the unlink's own mark once
the name is gone and the inode still has a path.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* mount: carry a link's stored entry into the file's open handle
Link rewrites the source entry with the new hard link id, count and times,
but an open handle kept its old base and entry. Its next conditional
content flush was refused (EIO), and an unconditional one would have
dropped the hard link id. Move the handle onto the stored entry when only
the link fields and times changed.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* mount: carry an alias unlink's stored entry into the open handle
Removing one of several hard links lowers the stored link count and
stamps a new ctime, and the handle may still hold the removed name. Look up
the entry at the name the handle flushes through and adopt it, so the next
content flush matches the stored entry, writes the right count, and
attribute saves address the surviving name.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* mount: test an open handle across link and alias unlink
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* mount: read an alias unlink's lost entry again at the next flush
When the read of the surviving entry right after an alias unlink fails,
the handle kept a base with the old link count and ctime, and every later
flush was refused even after the filer recovered. Leave the hard link id
on the handle and have the next flush read the stored entry and adopt it
before publishing.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* mount: publish a queued close before unlinking a name another link keeps
With writeback caching, close only queues the flush. Unlink marked the
handle deleted before waiting for that flush, so the flush skipped its
metadata and the surviving link kept the content from before the close.
When another link keeps the file, wait for the queued flush first.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* mount: let a queued close reach the filer before linking the file
With writeback caching a new file is created on the filer by its queued
close, and a listed directory serves the source from the cache, so a link
right after close updated a name the filer did not have yet and failed
with EIO.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* mount: skip a link change no newer than the handle's entry
A link change carried into the handle could replace a base already moved
past it by a later acknowledgment. Use the same clock-domain check as
installAckedEntry.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* mount: keep a link change's ctime in the dirty entry
The flush does not restamp times, so a dirty entry carrying an earlier
write's ctime moved the filer's ctime back from the link or unlink.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* mount: flush through the file's current name when the resolved one is gone
doFlush resolves the name before uploading the data. An unlink of that
alias, or a rename, in the meantime sent the publish to a name the file no
longer has.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* mount: keep the pending link read when its name is already gone
A flush could read the removed alias name, find nothing, and drop the
pending read of the surviving link, leaving the stale base in place.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* mount: test link and alias unlink across queued closes, flush races, versions and ctime
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* mount: pick a flush's name again once its locks are held
An unlink or rename that lands between the name pick and the locks would
otherwise send the flush to the removed name, where the create guard fails.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* mount: test a flush waiting on its lock through an alias unlink
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* mount: check the flush target after every lock acquisition
A flush whose name kept changing published to the name chosen before its
last lock wait. Check after each acquisition and return a retryable error
once the attempts run out.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* mount: test a flush whose name changes during every lock wait
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* shell: return flag parse errors from every command
48 commands returned nil when their flags failed to parse, so a mistyped
flag printed the flag error but the command still counted as a success:
a piped weed shell run exited 0, the master maintenance loop logged no
error, and admin script jobs reported the line as done.
Return the parse error as s3.bucket.create (#12544) and the other shell
commands already do. TestCommandsReturnFlagParseErrors runs every
registered command with an unknown flag and fails for the 48 on master.
Signed-off-by: anant <170962632+antcybersec@users.noreply.github.com>
* shell: don't report -h as a failure
With the parse errors returned, -h on those commands now returns
flag.ErrHelp, so the shell printed "error: flag: help requested" and a
piped run exited 1. The flag package has already printed the usage, so
processEachCmd now treats flag.ErrHelp as success for every command.
Signed-off-by: anant <170962632+antcybersec@users.noreply.github.com>
* shell: keep the -h test beside the flag parse test
TestProcessEachCmdTreatsHelpAsSuccess covers the same change as
TestCommandsReturnFlagParseErrors, so it lives in the same new file and
shell_liner_test.go stays as it was.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Signed-off-by: anant <170962632+antcybersec@users.noreply.github.com>
Co-authored-by: anant <170962632+antcybersec@users.noreply.github.com>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* credential: migrate the legacy IAM file without overwriting newer entries
The migration wrote every identity, service account and group from the
snapshot the load read, over whatever the per-entry files held by then, so
a revoke or delete landing in between was undone for good. It now migrates
only the legacy file's own entries, creates each per-entry file only when
none exists, and waits for the next load when the legacy file changed
after the snapshot.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* credential: test changes landing between the IAM load and its migration
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* credential: keep a per-entry file deleted after the IAM load deleted
An entry the load read from its own file is already migrated. When that
file is removed before the migration runs, the legacy copy must not
recreate it.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* credential: test a deleted per-entry file the legacy file also holds
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* s3tables: create a missing directory exclusively in ensureDirectory
ensureDirectory looked the path up and, when it was missing, created it with a
plain CreateEntry. A table create that lost the lookup race then replaced the
directory another create had made, claimed and confirmed since, erasing that
table's record, so a third create could claim the name and win as well.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* iceberg: create missing metadata parent directories exclusively
saveMetadataBlob created each missing directory on the way to the metadata
directory with a plain CreateEntry after a lookup. A table directory another
create had made and written its record onto since the lookup was replaced by a
bare directory, erasing that record.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* iceberg: create missing stage-create marker directories exclusively
The marker path starts at the table bucket directory, which carries the bucket
record; a plain create after a missed lookup replaces a directory another writer
made in between, with whatever it carries.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Volume layouts are keyed by ttl.String(), so a nil ttl and the empty ttl
share one layout. ttlEqual told them apart, so a replica reported with a nil
ttl into a layout created with the empty ttl matched its own layout as a
duplicate, and every copy was quarantined.
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* shell: replicate the downloaded volume before deleting its remote object
volume.tier.download downloaded every replica from the remote tier and let
the last one in the master's location list delete the shared object, so a
replica missing from that list kept reading a deleted object.
Download onto one replica, copy it over every other remote-backed replica,
re-read the topology so replicas that show up meanwhile are replaced too,
and only then delete the last remote-backed replica together with the
object before copying the local volume onto it.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* shell: test tier download replicates before deleting the remote object
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* shell: copy the downloaded volume onto each replica's own disk type
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* shell: test tier download keeps each replica's disk type
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* shell: name the hdd disk type when copying a downloaded volume
VolumeCopy treats an empty disk type as the source's, so an HDD replica
copied from an SSD source was placed on SSD or failed.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* shell: block deletes on the tier download copy source
A readonly-can-delete source took deletes after VolumeCopy read its
index, so the replaced replica lost them and served the files again.
Hard-mark the source readonly while replicas are replaced, then restore
the writable mark or reload the persisted state.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* shell: keep the remote object while replicas are unlisted
Tier download deleted the shared object once the listed replicas were
local, even when the replica placement expected more copies than the
topology listed. Keep it in that case so a rerun can finish later.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* shell: fail tier download when no replica is listed
Also point the final-replica delete error at volume.fix.replication.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* shell: keep the remote object when a server lists the volume twice
A stale twin on another disk of a replaced server still references the
object but is filtered out with its server.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* shell: restore the tier download source when its readonly mark fails
VolumeMarkReadonly blocks local writes and deletes before its second master
notification, so a failed notification returns an error with the source
already frozen. Restore it whenever the status was read.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* shell: report a failed tier download source restore
A restore that leaves the source unmounted or frozen was only printed, so the
command still returned success. Join the restore error into the result.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* shell: wrap the tier download restore error
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* volume_move: persist the move's source freeze across reloads
A live move froze the source in memory only, so an unmount/mount or a
volume server restart mid-move reopened it for writes that the move then
deletes with the source. Persist the freeze when the move froze a
writable source, and reopen the target right after the copy since the
copied .vif carries the freeze. An already-readonly source keeps an
in-memory mark, and a failed move still restores the source writable.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* test: live move keeps the source frozen across a reload, on Go and Rust volume servers
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* test: require 409 from the reloaded frozen source
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* volume_move: reopen an interrupted move's source only without a target copy
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* mount: merge a refused content flush onto the stored entry
A content flush conditioned on the handle's base failed with EIO whenever a
peer had changed the file since, so two mounts appending to one file could
not both close it cleanly. On a refused condition, reread the entry and
replay this handle's changes since its base onto it: the chunks it added or
dropped, and the attributes, extended keys and inline content it changed.
A local truncation keeps its size and clips the peer's chunks to it;
otherwise the larger size wins. The merged entry is published against the
stored one, up to three times, and installed in the handle so its next
flush keeps the peer's chunks. A replaced or deleted file is still refused.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* mount: merge a peer's content change into a dirty handle
A subscription event carrying a peer's content change replaced a dirty
handle's entry and discarded its unflushed pages, so the handle's writes
were lost and its close still returned OK. Fold the peer's entry into the
handle with the same merge a refused flush uses, keep the dirty pages, and
take the peer's entry as the new base. A replaced file still invalidates.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* mount: stop a flush merge once a filer accepts the write
withFilerClient can call the callback on several filers. A refusal from
one left its stored entry behind, so a later filer's success went on to
another merge round.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* mount: refuse a merge that puts inline content next to chunks
A read inside the inline bytes returns them without looking at chunks,
so one side's inline content merged with the other side's chunks hid
data after a successful flush.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* mount: refuse to merge a whole-file rewrite over a peer's chunk change
maybeMergeChunks re-uploads the handle's whole view as unstamped chunks.
Merged after a peer's change, they tie with a peer's own rewrite or bring
back a range the peer truncated, so that flush fails as before.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* mount: clip peer chunks at the lowest truncate, not the final size
A handle that truncated and then wrote past the old end ends up larger
than its base, so the merge did not see the truncate and kept the peer's
chunks inside the range the truncate cleared. SetAttr now records the
lowest size truncated to over the current base.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* mount: keep a peer's truncate when the handle left the size alone
The merge took the larger of the handle's and the stored size, so a peer
truncate came back as a zero-filled tail when the handle only overwrote
bytes inside it. The stored size now stands unless the handle changed
its size or wrote past it, and rebasing keeps only the growth writes made
after the flush was taken.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* mount: retry a read view whose merged manifests did not resolve
A failed SetChunks leaves the chunk group answering every read with the
resolution error, and nothing re-ran it while the entry stayed the same.
The handle now marks such a view (on rebase and on SetEntry) and the
next read or lseek resolves the entry's chunks again.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* mount: keep a dirty handle's writes when a peer change cannot merge
The new merge refusals sent a peer event on the same file down the old
path, which discarded the dirty pages and adopted the peer's entry, so
close returned OK without the writes. The handle now keeps its writes and
base, and the flush meets the conflict and fails.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* mount: keep an unpublished truncate across a peer's base change
The truncate mark was tied to the base pointer, so a peer event that moved
a dirty handle's base (a merged content change or a metadata-only change)
dropped it, and a later merge on a regrown file kept peer chunks inside
the truncated range. Those base moves now carry the mark; a flush ack or a
discarded handle state still clears it.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* mount: wait for the flush before merging a shrink over buffered writes
A write inside the file leaves its bytes in dirty pages without moving
the size. Merging a peer's truncate then adopted the smaller size, and the
chunk uploaded later sat past it, unreadable through the handle. The event
path and the periodic metadata flush now refuse a merge that shrinks the
handle; the close flush uploads the pages first and merges then.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* mount: tell a shortened base chunk from a whole-file rewrite
A truncate shortens a base chunk in place, so it gets a new reference
key, and an old chunk can carry no timestamp. The rewrite refusal now
applies only to unstamped chunks whose file id is new to the base.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* mount: check the creation time when the base has no inode
An entry without an inode or hard link id matched any replacement at the
same path, so a stale handle could merge into a different file. Without
an inode the merge now needs the same creation time. The filer keeps it
across updates; only a synthesized base has neither and still merges.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* mount: keep a dirty handle's writes when a peer replaces the file
A peer entry with another inode at the handle's path went down the
fallback that destroys the dirty pages and installs the replacement, so
close returned OK without the writes. The handle now keeps its writes
and base like it does for an unmergeable change, and its flush refuses to
update the replacement.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* mount: build the content flush tests on freebsd
The invalidation tests, which every platform compiles, now use the
content flush helpers, so the freebsd cross-compile vet failed with
undefined: flushConditionEntry. Nothing in the file is freebsd-specific.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* mount: keep a dirty handle's base when a same-content file replaces it
A replacement with the base's chunks but another inode counted as a
metadata-only change, so the dirty handle adopted it as its base and the
next conditional flush overwrote the new file.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* mount: bring a peer's metadata change into a dirty handle
A metadata-only event moved a dirty handle's base but left its live
entry, so the next flush read the peer's chmod as a local edit and
published the old mode back. Replay the handle's own changes onto the
peer's entry instead, as for a content change.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* mount: wait for the flush before merging peer inline content over buffered writes
A buffered write is not a chunk yet, so a merge could install a peer's
inline content that later reads return ahead of the write's chunk.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* mount: track a whole-file chunk rewrite on the handle
The merge told a rewrite by its unstamped chunks, which a manifest fold
hides: manifests are unstamped either way. Record the rewrite against the
base it was made from, carried across peer base changes like a pending
truncate, and refuse the merge from that.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* mount: refuse a merge that shortens a manifest at the truncate
Clipping a manifest reference leaves its members whole, so peer bytes past
the handle's truncate could reappear once the file grew again.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* mount: flush a dirty handle after a peer replaces its file
The replacement test checked only the retained handle state. It now
flushes against a filer holding the replacement, including one with the
base's content, and checks the flush fails and leaves it unchanged.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* mount: test an append fsync after a peer's attribute-only change
A second mount's acknowledged chmod, chown, utimes or setxattr leaves
the content alone, so the writer's conditional content flush must still
publish its append and keep that change. On master the refused flush
surfaces as EIO from Fsync.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* s3api: sign every operation subresource in signature V2
The V2 canonical resource only kept the subresources inherited from the
original allowlist, so tagging, retention, legal-hold, cors, encryption,
object-lock and the other keys the router dispatches on were left out of
the signature. A V2 signature issued for one request therefore also
verified that request with one of those keys appended, running a
different operation under the signer's identity.
Add every operation-selecting subresource to the signed list. Clients
that already sign tagging, cors, object-lock or replication in V2 now
verify as well.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3api: sign every value of a repeated signature V2 subresource
canonicalizedResourceV2 kept only the last value of a repeated query key,
while handlers read the first. A request signed for versionId=A could be
sent as versionId=B&versionId=A and act on version B. Sign every value in
order, as V2 clients do, so a second value no longer verifies.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* fix(mount): preserve acknowledged chunks during attribute updates
* mount: keep the WFS field block aligned
Adding the publication-lock fields inside the aligned block made gofmt
re-indent 25 unrelated lines. Put them in their own group after
dirIdleEvict, matching the enterprise layout, so syncs do not conflict there.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* mount: finish attribute saves through a FUSE interrupt
saveEntryAttributes derived its RPC context from the syscall's cancel
channel, so an interrupt while the conditional UpdateEntry was at the
owner returned EIO for an update the owner then applied, and skipped the
local install and retention. Keep the metadata deadline but drop the
cancellation, like the other metadata mutations.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* mount: map attribute CAS ids in place instead of cloning the entry
saveEntryAttributes cloned the whole entry three times per attempt only
to rewrite uid/gid for the filer, and each clone is as large as the
chunk list. Switch the ids in place and restore the exact local values
after the call, keeping one clone. The test now checks the reply carries
the local uid.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* mount: compare attribute updates against the filer's own uid/gid
With -map.uid/-map.gid, a filer id that is also a mapped local id reads
back unchanged and maps forward to another filer id. The IF_ENTRY_EQUAL
expected entry then never matched, so chmod, utimes and xattr changes on
such files failed with EAGAIN after three attempts, and an update would
have rewritten the owner. Keep the filer ids read with the entry for the
ids the syscall leaves alone.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* fix(mount): align attribute retention with read lock order
* mount: take the handle lock before entryLock in the attribute snapshot
attributeEntrySnapshot held entryLock while it queued for the shared
handle lock. A Read holding that lock waits on entryLock, the snapshot
waits behind a queued exclusive (an invalidation or a Write), and the
exclusive waits on the Read.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* fix(mount): complete handle lock order and preserve explicit ID intent
* mount: re-read a refused attribute update from the file's owner filer
The filer checks the IF_ENTRY_EQUAL condition on the owner of the path's
write route key. The retry read any filer, so behind one that had not
applied the owner's latest update, every attempt sent the same stale base
and the syscall failed with EAGAIN. Read from the owner once the lock ring
is known, like lookupFilerEntry, but keep the filer's raw ids.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* fix(mount): condition content flushes on the observed entry
* mount: stop retrying a flush the filer refused on its condition
The filer reports a failed CreateEntry condition as FilerError_PRECONDITION_FAILED
in the response, which surfaces as filer_pb.ErrPreconditionFailed rather than a
FailedPrecondition status. The flush retry loop treated it as transient and
replayed the refused flush three more times over 7 s before returning EIO.
The test filer now answers like the real one.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* mount: take the handle base from the entry the filer stored
After an acknowledged flush the handle base was the requested entry, but the
filer stores it with a fresh commit_ts_ns (and fills crtime, inode, atime), so
the next IF_ENTRY_EQUAL flush on the same handle never matched: write, fsync,
write, fsync returned EIO, as did close after a periodic metadata flush and the
first flush after an eager create. Use the stored entry from the
acknowledgement's metadata event.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* mount: condition the periodic metadata flush like the close flush
The periodic metadata flush of an open, dirty handle sent the same whole entry
through an unconditional CreateEntry, so a file held open past the flush
interval still overwrote a peer's acknowledged append, and the close flush then
matched the overwritten state. Both paths now publish through one helper that
carries the handle's write condition.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* mount: republish a refused flush once when the stored entry is unchanged
A handle's base can differ from the stored entry without any peer write: a
rename while the file is open restamps the entry under its new name, and with
-map.uid a filer id that is also a mapped local id does not map back to itself.
Each made every flush of the handle fail with EIO. On a miss, read the stored
entry and, if it still holds the observed state apart from name, commit stamp,
id encoding and derived size, retry once with that entry as the condition.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* mount: take the filer's entry once a write open holds the DLM lock
The lock orders this writer after the previous holder, but the holder's
flush may not have reached the local cache yet, so the handle's base
named the entry from before the wait and the conditional flush refused.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* mount: share the open-existing path of Create
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* mount: open the file a previous DLM holder created during a deferred create
A deferred create never asks the filer, so a create that waited for the
lock went on to publish over a file the previous holder had just made,
and its flush expecting no entry was refused.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* master: send the lock ring snapshot to mounts
A mount running -dlm is a lock client like the S3 gateway, and needs the
ring on connect to reach a file's owner filer before the next membership
change is broadcast.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* mount: follow the masters' lock ring under -dlm
The lock client only knew the seed filer, so every lock took the forward
hop and the mount had no way to tell which filer owns a file. Subscribe to
the masters named by the filer, like the S3 gateway, and wait briefly for
the first ring before serving.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* mount: read the DLM-ordered entry from the file's owner filer
The filer checks a flush's condition on the owner of the path's write
route key, and another filer may not have applied the previous holder's
flush yet. Look the entry up on that owner once the ring is known.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* volume(rust): untrack queued writes before replying
The durable write worker sent each write's result and only then dropped
the batch's pending-write trackers, so a caller could see its write
finish while the write was still tracked as pending. Go untracks in the
caller before writeNeedle2 returns. Drop the trackers before sending the
results so the Rust queue path matches.
test_replicate_delete_cancels_a_queued_write asserted the count was zero
right after the fsync write returned and failed about 1 run in 100 under
load (Rust Unit Tests on master c2625dd).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* volume(rust): untrack a write rejected for a remounted volume before replying
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* volume: skip VolumeDelete when its RPC was cancelled before the delete
A caller that cancels (a lost tiering lease, an expired deadline) still
had its admitted delete run in the Go server, including one that waited
behind the copy guard. The Rust server drops a cancelled handler, so
return the context error before the delete starts.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* volume: wait until the queued delete blocks on the guard before cancelling
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* s3api: verify signed request bodies against the signed payload hash
A SigV4 signature covers the declared x-amz-content-sha256, but only
getRequestDataReader (PutObject, UploadPart) compared the body with it.
Every other handler parsed the body as received, so a signed request,
DeleteObjects for example, could be replayed with a different body.
After a V4 signature verifies, wrap the body so it is handed out only
once all of it has been read and matched against the hash the signature
covered; a decoder that stops reading early never sees unverified bytes.
getRequestDataReader unwraps it and keeps verifying uploads as they
stream.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3api: answer XAmzContentSHA256Mismatch for refused DeleteObjects bodies
DeleteObjects answered a body that failed its payload-hash check with
InternalError and CompleteMultipartUpload with MalformedXML. Name the
check that failed instead, as PutObject does.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3api: check Content-MD5 on signed request bodies
Signature V2 signs Content-MD5 rather than a payload hash, and a V4
request sent with UNSIGNED-PAYLOAD may still sign it, but bodies read
outside the upload path never compared it with the body. Check it with
the payload hash once a V2 or V4 signature verifies: BadDigest on a
mismatch, InvalidDigest when it is malformed.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3api: verify presigned uploads against a signed payload hash
A presigned V4 URL can sign X-Amz-Content-Sha256 as a query parameter,
but getRequestDataReader compared upload bodies only with the header of
that name, so the signed hash went unchecked. Verify uploads against the
hash the signature covered.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3api: check Content-MD5 on bodies buffered for their payload hash
IAM and other non-S3 requests signed without x-amz-content-sha256 have
their body buffered and hashed during signature verification. That
buffer counted as already bound, so their Content-MD5 was never checked.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3api: check Content-MD5 on directory marker bodies
A PUT of a key ending in "/" stores its body on the directory entry
without going through putToFiler, so its Content-MD5 was never compared
with the body. Answer BadDigest on a mismatch, as putToFiler does.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3api: wrap and unwrap signed bodies through helpers
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* shell: confirm trim survivors still serve the volume before deleting
volume.fix.replication compared the doomed replica's index with every
survivor and then deleted it unconditionally. A survivor that went away
while the indexes were read still let the delete through, leaving fewer
copies than the check had counted. Re-read each survivor's VolumeStatus
after the index checks and skip the trim if any no longer has the volume.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* shell: test a trim whose survivor goes away after its index read
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* s3: ignore unsigned object headers on POST Object
The POST policy signs form fields only, but the upload read tagging,
user metadata, storage class, object lock and the stored content
headers from the request headers as well. Drop them before the form
fields are applied.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3: test POST Object request headers against signed form fields
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* s3api: answer PutObjectLegalHold on a missing key with NoSuchKey
In a versioned bucket a key with no latest version resolves to
ErrLatestVersionNotFound, which the legal hold handler did not map, so
the request failed with 500 InternalError. PutObjectRetention already
maps it to NoSuchKey.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3api: keep failed reads distinct when resolving an object lock target
getObjectEntryForUpdate turned every lookup error into a not-found
sentinel, so a failed filer read made PutObjectRetention and
PutObjectLegalHold answer NoSuchKey for an object that exists. Only a
genuine miss becomes the sentinel now, matching getObjectEntry.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3api: answer object lock calls on a delete marker with NoSuchKey
PutObjectLegalHold and PutObjectRetention on a key whose current
version is a delete marker returned 200 and wrote the lock onto the
marker, and the matching GETs read it back. A key whose current version
is a delete marker does not exist, and delete markers take no lock, so
these calls now answer NoSuchKey like the tagging and ACL handlers.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3api: let a lock recorded on a delete marker not block its removal
Delete markers are not WORM-protected, but a legal hold or retention
written onto one (possible before the previous change) made both the
gateway check and the routed delete's filer-side WORM guard refuse to
remove the marker, so the object could not be restored and lifecycle
could not clean the expired marker up.
Lock enforcement now ignores a lock on a delete marker, and the routed
version deletes in DeleteObject and lifecycle drop the WORM guard when
the version they read is a marker. A version id never changes kind, so
the marker read is the one deleted; the null version keeps its guard
because a PUT can replace it.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
The master, volume, volume-rust, server, mini, filer and s3 cases and the
fallback forwarded the user arguments as an unquoted $@, so the shell split
each one at whitespace and globbed it against /data. An -admin.password with
a space reached weed as two words, and weed ignored every flag after the
stray word. Quote "$@" as the other cases already do; $ARGS stays unquoted
since the script builds it from its own words.
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* replication: let the local sink check a delete against the live source
LocalSink removed the destination path for any delete it was handed. A
delete that reaches the sink after the source recreated the path (a peer
filer's event delivered late, or a historical event replayed) removed the
newer file, and no later event brings it back.
DeleteEntryVersion receives the deleted entry. Before removing an existing
destination, the local sink looks up the source path and keeps the
destination when the source holds an entry of the same kind committed
after the deleted version (commit_ts_ns, or mtime for unstamped entries).
A live entry that is the deleted version itself or older means the source
store lags the event, so the delete proceeds. A failed lookup returns the
error so the event is retried instead of deleting a file that may be live.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* filer.sync, filer.replicate: hand the deleted version to the sink
Pure deletes, renames without a native move and moves out of the
watched directory now go through sink.DeleteEntryVersion with the old
entry, so a sink that checks the live source (the local sink) can keep a
path the source recreated after the deleted version. Other sinks fall
back to DeleteEntry unchanged.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* replication: order the local sink check by the delete's mutation time
A source filer can hold a version written concurrently with the delete:
newer than the deleted entry but older than the delete itself. The
source's own replay applies the delete over it, so keeping the backup
copy because it outranks the deleted entry left a file the source no
longer has.
The sink now gets the delete's mutation time (MutationTsNs, else TsNs)
and keeps the destination only when the live entry was committed after
it, the order Filer.Replay keeps. That also covers a stamped replacement
of an unstamped entry whose mtime did not move. filer.replicate events
carry no time, so they keep the deleted entry's stamp as the bound, and
mtime only when nothing is stamped.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* replication: count an unstamped live entry from its mtime
A source filer that predates commit stamps stores entries with no
CommitTsNs, so comparing the live stamp with the delete's time never
kept a replacement there. An unstamped live entry now counts from its
mtime against the delete's time.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* replication: keep a stamped live entry when the deleted one is unstamped and the delete has no time
filer.replicate gets no delete time from the notification queue. When the
deleted entry has no commit stamp either, a stamped live entry was written
after it, so its mtime no longer decides.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
The line parser only trimmed quotes, so `key = "abc\n"` gave the Rust
volume server the bytes `abc\n` with a literal backslash while the Go
servers sign with a newline, and every JWT failed. Literal strings kept
their quotes and a trailing comment stayed part of the value.
Decode each value with the TOML parser; text it rejects keeps the old
quote trimming.
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* operation: probe octet-stream uploads for gzip like untyped ones
The 128-byte gzip probe ran only when the caller passed no MimeType, so an
upload labelled application/octet-stream or binary/octet-stream was never
compressed. S3 PutObject and UploadPart pass the request Content-Type (or the
octet-stream that mimeDetect fills in), so every binary S3 upload skipped
compression that the same bytes get through the filer.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* operation: check the probed upload keeps the caller's content type
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* wdclient: keep a shared volume lookup alive when its first caller leaves
Concurrent misses for the same volumes share one provider lookup, but it
ran on the first caller's context, so that caller's cancellation failed
every request waiting on it. The lookup now runs on a context no single
caller can cancel, and each caller stops waiting when its own context ends.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* wdclient: wait for the second caller to join before canceling the first
A fixed sleep did not prove the second caller had joined the shared lookup:
a late caller could hit the filled cache and pass without waiting. The test
now waits until the caller reads its own Done channel, which happens only
after DoChan registered it.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* filer: let the fake-master test helper serve any master
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* filer: gzip metadata log pieces before uploading them
Every metadata event carries the old and new entry with their whole chunk
lists, so a file grown by small fsynced appends logs the same chunk records
again and again: 2000 appends of 256 bytes through weed mount wrote 34.8 MB
of log. The uploader only compresses when the first 128 bytes compress, and
a log piece starts with fresh file ids and ETags, so it stored the log as is.
The filer now gzips each piece itself and keeps the compressed bytes when
they save at least 10%. After the same workload the volumes hold 3.05 MB
instead of 35.2 MB, file data included. Readers already honor the chunk's
IsCompressed flag on every path (filer-side decode, log file refs read
straight from volume servers), and the assign request sizes the bytes
actually stored.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* mount: keep the filer status in the flush create error
flushMetadataToFiler formatted the CreateEntry error with %v, so callers
classifying it saw only the message, which includes the file path. Wrap
it so the gRPC status the filer sent stays reachable.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* mount: retry a writeback metadata flush until the filer answers
With -writebackCache, close() returns before the chunk references reach
the filer. The deferred flush gave up after four attempts, about a minute
once each attempt's own filer retries are counted, then released the
handle: a longer filer outage dropped the file even though write() and
close() had succeeded, and its uploaded chunks became orphans.
Keep retrying for as long as the filer is unreachable, as the writeback
chunk uploads already do. A filer that answers with an error still ends
the flush after the bounded attempts, and an unlink of the draining file
still ends it on the next attempt.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* mount: retry only an unanswered filer call without a bound
The unbounded phase used util.IsTransientError, which falls back to
matching the error text. A refusal the filer sends in its response
(read-only, name too long, a plain error string) has no gRPC status, and
the flush error carries the file path, so a file named like
"unavailable-transport.log" kept retrying against a filer that answered.
Retry without a bound only when the call failed with a gRPC Unavailable
status: the dial, the connection or the mount's call deadline gave out.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* s3api: deny signed reads that name a version file by its physical path
A stored version lives at <key>.versions/v_<id>, and that path is also a
valid object key. Anonymous reads already refuse it; signed GET, HEAD,
copy sources and object subresources did not, so a caller without
s3:GetObjectVersion, or under a Deny on the logical key, could read any
version through it. Apply the same check in the shared entry recheck.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3api: recheck a version file key before answering a conditional read
A conditional GET or HEAD answered 304 or 412 with the ETag before the
handler's entry recheck, so the physical version path still confirmed a
version's ETag to signed callers.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3api: look up version file keys in the lookup-based policy recheck
Retention, legal hold, tagging writes and deletes only fetched the entry
for the recheck when the bucket had a policy, so without one they still
acted on a version file named by its physical path. Fetch it whenever
the key has the version file shape.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3api: refuse a directory-marker delete that names a version file
A trailing slash kept "<key>.versions/v_<id>/" out of the version file
shape check, and the lookup recheck misses the entry for that form, so a
DELETE reached deleteDirectoryMarker and removed a chunkless version or
delete marker without s3:DeleteObjectVersion.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
The ring tests used 127.0.0.1:18889 as an unreachable peer, but its gRPC
port 28889 is a common filer port: on a shared CI host a live filer there
accepted the forwarded exclusive create, and
TestCreateEntryExclusiveLeavesNonOwner failed. The peer now listens on a
port the test owns and drops every connection.
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* vector: share one Lance session across reads
Each request opened its table in a session of its own, so every query
loaded and decoded the index again. Get, list and query now open in one
session the server keeps, sized by -index-cache-mb and
-metadata-cache-mb; writes keep a session of their own.
Lance keys some of its cached entries, such as a version's index list,
by location and version alone, and an index dropped and created again at
its location numbers its versions from 1 again: a shared session answered
a query on the new table with the old table's index. CreateIndex now
marks each table with an id in its schema metadata, and the session
starts over when a location turns up holding another table.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* vector: share a version lease among the reads of that version
Every get, list and query tagged the version it read and deleted the tag
afterwards: two filer writes and a manifest read per request. Reads of
one version now share one tag, which goes 10 s after the last read took
it; a tag is reused for at most 5 minutes, so a busy version never
reaches the reapers' TTLs.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* vector: keep opened tables for the next read
A read built the table's store and read its latest manifest each time.
Reads now keep the opened table by location and encryption, up to 1024,
and move it to the latest version, which lists the versions and takes
the manifest from the session. A handle opened in an earlier session is
not reused.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* vector: key shared leases by manifest, and take one per version at once
A table dropped and created again within the linger reused, for a
version number the old table had, a lease whose tag the drop had
deleted. Leases are now keyed by the manifest's ETag as well. Reads that
arrive before a version has a lease wait for the first to take it,
rather than each writing a tag.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* vector: drop kept tables when the shared session starts over
A kept table holds the session it was opened in, so a replaced session
and its caches lived on in the handles of other tables until they were
evicted.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* vector: prune shared lease slots only when they double
Every lease lookup scanned all slots under the map's lock, a cost that
grew with the versions read in the last linger.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* vector: shrink the lease slot map after a prune
After a burst the map kept its peak allocation, which every later prune
scanned.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* vector: keep a lease slot a read has taken through a prune
A read that took a slot and had not yet locked it could see the slot
pruned, and then pin through it while the next read pinned through a
new slot, writing a second tag.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* admin: restyle the UI as a calm console
Grey canvas with white sheets, a docked sidebar that marks the current
page with an indigo tint, ink primary buttons, tinted status pills,
dense tables with tabular numbers and spectrum capacity meters. The
styles are design tokens layered over Bootstrap's own variables, with
light and dark values, so the page templates stay unchanged.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* admin: add a light, dark and automatic theme switch
The choice is kept in localStorage and applied before the page paints,
on the sign-in page too; automatic follows the system setting.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* admin: drop hard-coded colors from page templates
The volume id link, utilization track and plugin form borders used
fixed light-theme colors; sparklines now use the data palette.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* admin: theme the plugin job detail panel background
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* admin: keep the selected radio dot visible in dark mode
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* admin: cycle the theme from memory when storage is blocked
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* admin: keep success progress bars green
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* admin: keep keyboard focus visible on the toolbar togglers
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* admin: use contrasting text on danger and success buttons in dark mode
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* admin: lead the dashboard with cluster health and one row of tiles
The header now says in one line whether the cluster is healthy and how
many masters, volume servers and filers report. The two rows of stat
tiles become one: the volume size limit and EC volume count move into
footnotes, the mount client card joins the cluster counts, and the
counts link to their pages instead of sitting in nested cards. The
duplicate version list and the bucket shortcut in the header are gone,
and the section menus become plain View all links.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* admin: export the user columns the table actually has
The users table has username, email and access key columns, but the
export waited for six cells per row and so always produced only the
header line.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* admin: drop table titles that repeat the page title
List pages titled their only table with the page title again. The
table now sits right under the summary tiles, and the export actions
that were hidden in the table header menu move to the page header as
an Export button, matching the cluster pages. The policies menu linked
to nothing and is removed.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* admin: drop clock and duplicate summary tiles
Buckets, users, policies and service accounts spent a summary tile on
the refresh time, which the page footer already shows (policies gains
that footer). The users page also showed its user count twice.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* admin: show unset bucket settings as a dash
Rows of No owner, No quota and Not configured made the bucket table
hard to scan. Unset cells now show a quiet dash that keeps the wording
as a tooltip and for screen readers, and the name and creation date no
longer wrap (the full timestamp is in the date's tooltip).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* admin: export buckets through the page's API exporter
admin.js loads after the page, so its copy of exportBucketList replaced
the buckets page's own one. That copy scraped the visible rows, so it
exported only the current page, without CSV quoting, and picked up the
display text of the cells. The page's exporter reads every bucket from
the API with full timestamps and escaped fields.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* admin: escape quotes in the user export
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* admin: download the bucket export as a Blob
The exporter put the CSV into a data URL with encodeURI, which leaves #
alone, so a bucket owner containing # ended the download there. Use the
shared downloadCSV helper instead.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* admin: count masters as configured, not reporting
The master list comes from raft membership, which keeps a follower
that is down, so it does not show which masters are reporting.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* admin: link the Prometheus metrics endpoint from the sidebar
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* admin: drop the sidebar metrics link
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* s3api: record the caller on AssumeRole sessions
The session name is the caller's choice; the session now carries the
caller itself as the user it was issued to, as the web identity, LDAP
and credential paths already do.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3api: take a role session's aws:username from its user, not its name
An assumed-role ARN ends in the session name the caller picked, so
naming a session after another user resolved ${aws:username} and
${aws:userid} to that user in bucket and attached policies. Resolve
them from the user the session was issued to instead.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* iam: take a role session's aws:username from its user, not its name
IsActionAllowed set aws:username to the last segment of an assumed-role
ARN, the caller-chosen session name, for role and bucket policies. Use
the user the session was issued to, and drop a forwarded value when the
session names none.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3api: pass a role session caller's user on to the session it assumes
A role session that calls AssumeRole is named by its session ID, so the
new session recorded that ID as its user. An STS session caller now
passes on its own recorded user, or none.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3api: size the session claims map from the request context
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* admin: take master HTTP ports from raft member names
The masters page and dashboard turned each raft member's gRPC address
into an HTTP address by subtracting 10000 from the port, so masters with
a custom -port.grpc showed wrong or negative ports. Raft members are
named after their master address, which carries the HTTP port; use it
and keep the old derivation only for names that are not addresses.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* admin: test raft member address mapping
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* s3api: drop cached gRPC channels when a fake filer takes a port
A test that dials a dead owner leaves its failed channel in the
process-wide gRPC connection cache. When a later test's fake filer is
given the same port, its first RPC fails at once with the cached
connection refused, which is how TestRoutedMultipartFinalize failed on
the 32-bit job. The fake filer helpers now share fakeFilerAddress, which
invalidates any cached channel for the new address.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3api: test a fake filer on a dead owner's reused port
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3api: drop cached channels for dead test filer addresses too
closedFilerAddress can be handed a port whose cached channel belonged to
a fake filer that just stopped and may still look ready; then the dead
owner's transaction is sent on it instead of failing over. Build its
address with fakeFilerAddress as well.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* s3api: split upload ACL assembly out of preparePutObjectACL
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3api: store POST Object acl and grant fields on the object
The form acl and x-amz-grant-* fields were authorized but never assembled,
so the object kept no ACL: acl=private on a public-read bucket stayed
anonymously readable and invalid values were accepted. Build them with the
same rules as PutObject, which also covers the Block Public Access check.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3api: test POST Object stores its upload ACL
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3api: take the POST Object ACL from the signed form only
ACL request headers are not covered by the POST policy, so a holder of a
signed form could add one and have it stored with the upload.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3api: reject a POST acl field the policy does not cover
The acl form field now sets the stored object ACL, so like the x-amz-*
fields it needs a policy condition, as AWS requires.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3api: test unsigned ACL headers and uncovered acl fields on POST
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3api: evaluate POST upload policies on the canonical ACL
Policy conditions on the grant headers saw the raw form value, so an
equivalent spelling of a denied grantee passed. Use the PutObject request
shape that the PUT path evaluates.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3api: check bucket policy for POST ACLs with auth disabled
PutObject applies the bucket policy to an explicit upload ACL when IAM is
off; POST now does the same before storing it.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3api: reject differing acl and x-amz-acl POST fields
Both promote to the canned ACL header, so map order picked the stored ACL.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3api: test POST grant canonicalization, auth-off ACL policy, acl conflicts
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3api: check the auth-off POST ACL policy after loading the bucket
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3api: reject an empty acl or x-amz-acl POST field beside a differing one
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3api: test the auth-off POST ACL policy on a cold bucket cache and empty acl aliases
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3api: load the bucket before per-key object authorization
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3api: test the POST ACL policy on a cold bucket cache with auth on
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3api: stop per-key object authorization when the bucket fails to load
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3api: test the cold-bucket POST ACL allow path and a failed bucket load
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* lance: require UpdateTable to deregister a table
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* lance: require UpdateTable to register over an existing table
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* lance: test that GetTable alone cannot deregister or re-register a table
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* lance: a register that finds a table the caller cannot read needs UpdateTable to bring it back
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* lance: decide declare write access by whether the table existed, not by what the caller can read
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* lance: test register and declare by a caller that cannot read the table
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* lance: register clears the marker on the create branch only after checking it
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* lance: keep a register whose table was created when the marker check fails
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* filer/leveldb_handle: reopen a database whose journal latched a write error
goleveldb keeps the first journal write error and returns it for every
later write until the database is reopened, so one transient ENOSPC or
EIO fails all writes until a restart. The handle offers the leveldb.DB
methods the stores use; a write that fails on an OS error probes the
storage and, once it accepts writes and no call holds the database,
closes and reopens it and retries the write once.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* filer: open the leveldb stores through leveldb_handle
leveldb, leveldb2 and leveldb3 now recover from a transient storage
write error without a restart.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* filer: leveldb stores return iterator errors from listings and deletes
A listing on a database that cannot be read reported an empty
directory, and DeleteFolderChildren wrote an empty batch, because the
iterator error was never checked.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* filer/leveldb_handle: test a reopen that fails after the probe passes
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* filer/leveldb_handle: probe storage before draining readers
A probe that stalls in Create, Write or Sync held mu exclusively and
blocked every read on the handle. Reopen attempts are now serialized by
their own mutex, and the probe runs before the drain.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* filer/leveldb_handle: hold new calls back while a reopen drains
TryLock succeeds only at an instant with no reader, so overlapping
listings could keep a reopen from ever draining while each write spun
for the full drain timeout. The drain now holds new calls back for at
most drainTimeout; a call nested inside a running one waits that out
instead of deadlocking. A drain that times out holds off the next
attempt for busyReopenInterval, so new calls are not held back over and
over.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* filer/leveldb_handle: bound the wait for a running reopen
A write that hit the latched error waited on the reopen mutex for as long
as another caller's storage probe ran, so a stalled probe queued every
writer. Callers now wait at most reopenWait and return their error, and
still retry once a concurrent reopen succeeds.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* filer/leveldb_handle: tests wait for the drain and readers explicitly
The nested-read test polled for a drain that could start and end between
polls and then spun forever; it now syncs on the probe, widens the drain
and fails if the put returns first. The overlapping-readers test waits
until every reader holds an iterator before it writes.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* s3: authorize an empty versionId as the current-object action
GET, HEAD and DELETE treat an empty versionId as the current object, but
the action resolver keyed on the parameter's presence and authorized it
as GetObjectVersion/DeleteObjectVersion, so a Deny on GetObject or
DeleteObject did not apply.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3: test an empty versionId against a version-only policy
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3: cover a bare versionId on HEAD
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* volume: refuse a tier upload whose .dat grew during the copy
AttachRemoteDatFile only checked that the data backend was still the one
copied, so a needle write acknowledged after the copy read the .dat was
missing from the remote object, and the upload then switched reads to it
and removed the local .dat. Compare the .dat size with the copy under the
same data-file locks; on a mismatch the handler deletes the copy and keeps
the local file.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* seaweed-volume: refuse a tier upload whose .dat grew during the copy
Mirror the Go AttachRemoteDatFile check. The tier-up attach took the
write lease only inside each step, so a needle write acknowledged after
the upload read the .dat was missing from the object the volume then
switched to, and the local .dat holding it was removed. Hold one write
lease across the instance, revision and size checks, the remote-file
update, the .vif save and the backend swap, as tier-down does; a size
mismatch aborts, deleting the object and keeping the local .dat.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* volume: compare the physical .dat size before a tier attach
DiskFile pads its size past an unaligned tail, which a read-only load
keeps, while the upload copies the physical bytes. Stat the file itself
so such a volume is not refused as written during the copy.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* shell: parallel moves and a per-run limit for volumeServer.evacuate
-maxParallelization runs up to N EC shard moves at once (default 1), with the
ec.balance scheduler: never two shards of one volume at a time, free workers
go to the least busy targets. The moves are planned first with the same target
choice as before, then applied through volume_move.Mover.MoveEcShards, which
verifies the target registered a shard before deleting it on the source.
-movesPerExec limits how many volumes and EC shards one run moves (default 0
for unlimited), like -volumesPerExec in volume.balance, so a big server can be
emptied in several runs.
With -skipNonMoveable a shard that fails to move (a read error on a failing
disk, for example) no longer drops the other shards of its volume; they are
moved and the failures are listed at the end. Without it the run stops at the
first failure as before. Losing the admin lock stops the run in either mode.
Refs #12501
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* shell: plan evacuation again after failed moves; fail on a lost lock
A failed shard move kept the target slot it was planned on, so a readable
shard that found no destination in the plan stayed on the server although
the room was never used. Failed moves now give their targets back to the
model and the shards without a destination are planned again, up to
maxEcEvacuationRounds times; the failed shards are not tried again.
With -skipNonMoveable a lost admin lock was printed as a skipped failure and
the run returned success. ec.ErrLockLost is now exported and always fails
the run.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* shell: volumeServer.evacuate reports -movesPerExec after an applied EC run
The round loop only checked the limit after a failed move, so an applied run
that stopped at -movesPerExec without failures never said to run it again.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* shell: volumeServer.evacuate plans again for as long as failed moves leave room
The ten-round cap left readable shards on the server once more than ten
rounds of unreadable shards had taken the free room first, and a rerun
fails the same shards first again. Every round moves or fails each shard it
plans and a failed shard is not planned again, so the rounds end without it.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* shell: volumeServer.evacuate gives a failed move back every slot it took
When the target already held the shard in another disk type, adding it to
the planned disk type still took a slot, but the undo gave back only the
explicit one. Record the slots the plan took and restore exactly those.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* ec: an evacuation move that landed on its target keeps the target slot
A move can fail after the target registered the shard, when only the source
unmount or delete failed. Evacuate then gave the target's slot back and could
plan another shard onto a full server. The runner now checks the target after
a failed move and leaves a shard the target has out of FailedMoves.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* shell: volumeServer.evacuate gives back the slot of a failed move onto a shard the target had
A target that held the shard before the plan reports it after any failed
move, so the runner took such a move for one that landed and evacuate kept
the slot the plan had taken. The runner now lists those moves in
LandedMoves, and evacuate undoes the ones whose target had the shard, then
plans again with the room they gave back.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
* util: add WaitUntil, a cond wait bounded by a context and a timeout
The caller holds the cond lock. Cancellation and the timeout wake the
waiter through a broadcast taken under the same lock, so no wakeup is lost.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3: bound the wait for an upload slot
A PUT that arrived while the concurrent upload limits were exhausted waited
on the condition variable with no bound. The handler reads nothing from the
body while it waits, so a client that gave up and reset its connection went
unnoticed: the socket and goroutine stayed pinned until a slot freed, and a
stalled volume server grew the gateway's fd count by one per aborted PUT.
The wait now ends on InflightUploadDataTimeout with 503 SlowDown, which S3
SDKs retry, and on request cancellation. An admitted upload takes its slot
under the lock, and a finished upload broadcasts, so freed bytes admit every
waiter that fits instead of one.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* filer: bound the wait for an upload slot
The filer's PUT/POST upload limits queued requests the same unbounded way.
Reuse the bounded wait: reject with 429 after InflightUploadDataTimeout, as
the volume server does, and stop on request cancellation.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* command: add -inflightUploadDataTimeout for the S3 gateway and the filer
weed s3 -inflightUploadDataTimeout, weed filer -inflightUploadDataTimeout and
-s3.inflightUploadDataTimeout, weed server and weed mini
-filer.inflightUploadDataTimeout and -s3.inflightUploadDataTimeout. The
default of 60s matches the volume server's flag of the same name; 0 keeps
the old unbounded wait.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3: reject a queued upload without reading its body
WriteErrorResponse drains the request body before answering. Behind the
stats recorder the drain has no read deadline, and reading the body sends an
Expect: 100-continue client the go-ahead to upload what is about to be
discarded. Answer the rejection with an empty-body copy of the request so
the drain is skipped; net/http closes the connection when a large unread
body remains.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3, filer: release an upload slot under the cond lock
A waiter checks the counters under the cond lock and then waits. A release
that decremented and broadcast without the lock could land between the two,
so the waiter slept with a slot free until the next release or its timeout.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3, filer: close the HTTP/1 connection when rejecting a queued upload
net/http reads up to 256 KiB of an unread request body before sending a
keep-alive reply, so a client that stalls its body held back the 503 or
429 until the connection idle timeout. Connection: close skips that read.
HTTP/2 is left alone: there it would shut down the whole connection.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* shell: s3.clean.uploads deletes uploads over filer gRPC
Same DeleteEntry the HTTP handler runs, without minting a filer JWT,
and the gRPC path is the one a locked, routed delete needs next.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* shell: s3.clean.uploads frees an open upload under the object write lock
The cleaner deleted a stale upload recursively with no lock, so a
CompleteMultipartUpload could pass its upload-exists check after the
part entries and their chunks were gone, and publish an object over
freed chunks. Do what AbortMultipartUpload does: hold the object's
distributed write lock, and free an open upload in an ObjectTransaction
routed to the object's owner filer, conditioned on the object not
carrying the upload; if it does, delete the metadata only.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* shell: s3.clean.uploads keeps the object write lock for the whole cleanup
A large upload's recursive delete can outlast one 15 second lease, and
a completion on the distributed-lock path could then take the lock and
commit over chunks being freed. Renew the lock in the background and
skip the delete once it is lost.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* shell: s3.clean.uploads locks the path a completion commits under
For a trailing-slash key a non-versioned completion writes and locks the
entry inside the directory the key names, not the object path, so the
cleaner's transaction did not serialize with it. Lock and check the
same path the completion uses for the bucket's versioning state; the
route key stays the object's.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3api: abort locks the path a completion commits under
A routed abort locked and checked the object path, but for a
trailing-slash key a non-versioned completion commits under the entry
inside the directory the key names. Use the same path the completion
locks for the bucket's versioning state.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* shell: end s3.clean.uploads bucket failure line with a newline
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* cluster: NewBlockingLongLivedLockWithin bounds the wait on a held lock
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* shell: s3.clean.uploads skips an upload whose object lock stays held
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* cluster: share the contended lock filer setup across lock tests
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* shell: s3.clean.uploads reads the bucket's versioning state per upload
The commit path a routed completion locks depends on versioning for a
trailing-slash key. Reading it once per bucket before the run left a
window as long as the run; read it just before the delete, as the
gateway's abort does.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* mount: remove the local control socket when the mount exits
weed mount listened on /tmp/seaweedfs-mount-<hash>.sock and removed it only
when a later mount of the same directory started, so every unmount or
SIGTERM left one socket file behind. Remove it on the normal return path and
in the interrupt hook, unless a later mount of the same directory has
already replaced it.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* mount: remove the control socket before a failed mount exits
glog.Fatalf and glog.Exitf call os.Exit, which skips the deferred removal,
so a mount that failed after creating its socket (mount point not
available, fusermount failure) still left it behind.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* mount: register the socket cleanup hook before mount setup
A SIGTERM that arrived while the mount was still setting up, for example
waiting on the filer to create the mount root, exited through grace without
running the deferred cleanup. Registering the hook right after the listener
is created makes it run last, after the unmount and filesystem hooks.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* filer/etcd: scan entries from one revision
volume.fsck walked etcd one directory at a time, so a file moved from a
directory not yet listed into one already listed was seen under neither
name and its chunks were reported as orphans. The etcd store now
implements filer.EntryScanner: every page is read at the revision of the
first read, and a compacted revision fails the scan instead of falling
back to the walk.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* filer/etcd: read hard links at the scan revision
Hard-link records resolved during a scan were read at the current
revision, so a link updated meanwhile mixed two instants. Reads made with
the scan's context now use its revision.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* shell: list etcd among the stores volume.fsck reads from a snapshot
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* filer/etcd: skip hard-link records in a scan
Hard-link records live beside the entries under their random ids, so a
root scan could read one as an entry with a path made of random bytes,
which the traversal stream cannot send. Skip hard-link ids and keys whose
directory is not valid UTF-8.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* filer/etcd: recognize a hard-link record by the id it stores
A hard-link id can be valid UTF-8, so the key shape alone let some
records through a root scan as phantom files. The record's value is the
shared entry, whose HardLinkId is the record's own key; no entry key
can equal its own hard-link id.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* filer/etcd: scan entries under an invalid-UTF-8 directory
The filer sanitizes names but stores directories as given, so a path
written over HTTP can leave entries the listing never reaches. Skipping
them let volume.fsck count their chunks as orphans. Send them under the
sanitized directory instead, the substitution the filer uses for names.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* filer/etcd: fail a scan whose sanitized path names another entry
An entry under an invalid-UTF-8 directory is reported under the sanitized
directory. When that path is also another stored entry at the scan revision,
or another sanitized entry in the same scan, acting on the reported path
(volume.fsck deleting it) would reach the other file, so the scan fails and
names the stored path instead.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* ec: don't index ev.Shards[0] when the last shard was unmounted
* ec: skip a runtime emptied before the scrub merge reads its shards
FindAllEcVolumes releases each location lock before MergeEcRuntimes runs,
so an unmount can leave a merged runtime with no shards and the width
scan indexed Shards[-1].
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* ec: treat a scrub merge with no shard left as a vanished volume
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* ec: read Shards once in ShardSize so a concurrent unmount cannot empty it
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* ec: read each runtime's Shards once in the scrub merge
The vanished check and the slot construction read Shards separately, so
an unmount between them produced a non-nil view with no shards.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* ec: anchor the scrub merge on a generation that still holds shards
An unmount can empty the newest runtime while an older one keeps its
shards, which left a merged view anchored on the emptied generation with
no slots and the holder reported as skipped.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Mansur Akhmadi <10739460+mansur-akhmadi@users.noreply.github.com>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* shell: fs.mergeVolumes leaves in-progress multipart parts in place
CompleteMultipartUpload reads the part entries and later commits the
object with their fids, guarded only by the upload directory. A merge
that moved a part chunk in between deleted the source needle the new
object pointed at. Parts stay where they are until the upload completes
or is cleaned, and their chunks still count as referenced sources.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* shell: note in fs.mergeVolumes help that upload parts stay in place
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* shell: match upload parts when the buckets folder is the root
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* shell: count chunks behind a part's manifest as referenced sources
A skipped part whose chunks on a merge source sit behind a manifest was
not counted, so the merge printed no skip line and warned that nothing
references the source.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* shell: keep a part's direct source references when its manifest is unreadable
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* shell: count chunks resolved before an unreadable part manifest
ResolveChunkManifest returns the data chunks of earlier manifests along
with a later manifest's error; record them before returning so the
unreferenced-source warning does not fire for a source they reference.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* shell: test the partial manifest result without a timing delay
The unreadable manifest is now nested under a readable part manifest, so
the first level resolves fully before the failing lookup runs.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* filer_pb: add a depth-first traversal that can resume after a path
TraverseDfs walks a tree in listing order and reports each finished
path, so a caller can save one path and continue from it later. Pages
are read whole before visiting, and the entries of a page are visited
by up to five workers.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* filer.meta.backup: resume an interrupted metadata copy
The walk saves the last finished path every few seconds, and a walk
that continues an unfinished one starts after it. A fresh walk clears
the saved path first, and a finished walk clears it with the walk
start.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* filer.meta.backup: add -entriesPerSecond to throttle the metadata copy
Each entry the full copy reads waits on a rate limiter, and pages are
listed only after the previous page is copied, so the flag also bounds
how fast the copy lists the source filer. 0 keeps it unlimited.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* filer_pb: report each file of a depth-first walk once all before it are done
A throttled walk only reported progress after a whole listing page, so a
slow copy could go many minutes without a resumable path. Visit entries in
runs that end at a directory and report each file as soon as every entry
before it is done.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* volume: fail a vacuum step that finds a compaction already running
Compact, CompactByIndex and CommitCompact returned success when another
compaction held the volume. A second, uncoordinated vacuum (an expired
admin job still running, a shell volume.vacuum next to the worker) then
went on to commit or clean up the first run's .cpd/.cpx. Return a
"volume N is compacting" error instead, and refuse cleanup while a copy
is in flight, as the Rust volume server already did.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* volume: check the compaction files before a commit closes the volume
CommitCompact closed the needle map and data file and wrote the .cpc
marker before it found .cpd/.cpx missing. A duplicate commit, such as
the second of two overlapping vacuums, then left the volume registered
with no open files: it dropped out of the heartbeat and looked lost to
volume.list and volume.fix.replication until a restart. Check for the
files first so the refused commit leaves the volume as it was.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* volume: run makeupDiff before a commit closes the volume
CommitCompact closed the needle map and data file before makeupDiff, so
a makeupDiff failure (a compaction revision another commit moved, an I/O
error) returned with them closed: the volume stayed registered but
missing from the heartbeat until a restart. makeupDiff reads the live
files through its own handles while the commit holds the write lock, so
close them only once it has succeeded, as the Rust volume server does.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* volume: log a volume the heartbeat skips because its files are closed
A volume left without its needle map or data file vanished from the
heartbeat with no log line, so the master saw a replica disappear while
its files were still on disk. Warn on each skip, as the deleted-.dat
case next to it already does.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* volume: claim the compaction flag while cleanup removes the temp files
Cleanup only read the flag, so a compaction could start between the
check and the removal and lose its .cpd/.cpx. Claim it for the cleanup
instead, in Go and Rust.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* filer.sync: move the watermark over jobs that finished ahead of an older one
The watermark only took the finishing job's timestamp when it was the
oldest in flight, so jobs that completed before an older one were never
recorded and a quiet stream kept the offset behind finished events.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* filer.sync: keep one finished timestamp per gap behind a running job
A long job kept every later completion in the done heap. Only the newest
one between two running or failed timestamps can become the watermark.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* filer.sync: test done-heap compaction across several running and failed jobs
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* s3: answer GETs canceled before the first byte
A stream that fails before writing anything now always gets an S3 error
response, canceled errors included. The idle timeout and a shared lookup
canceled by another request both cancel the request context while the
client is still connected, and returning silently sent that client a
200 with Content-Length: 0. A client that has gone away never sees the
write.
Refs #12500
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3: mark inline GET write errors as written; bound the idle-cancel test
An inline object's write error comes after the status line, so the
caller must not write an S3 error on top of it. The idle-cancel test
caps the client wait and gives the response write its own deadline,
keeping only the read side on the 200 ms idle timeout.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3: test a GET whose request is canceled before the first byte
Drives GetObjectHandler through a chunk lookup that cancels the request
context, so a handler that returns before answering shows up as the empty
200 again.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3: check the cancel test reached the chunk lookup
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
* volume: forward no Content-MD5 for a JPEG the orientation fix rewrote
With -images.fix.orientation the primary stores and forwards the rotated
bytes, but sent replicas the client's digest of the original upload. The
replica checks Content-MD5 against what it receives and rejects the write,
so replicated JPEG uploads with a digest failed. The Rust volume server
already drops the digest once the body is rewritten.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* volume: test the forwarded Content-MD5 through PostHandler and a replica
The test now posts each upload to a primary whose fake master names a
real replica volume server, so it covers the JPEG gate, ReplicatedWrite
and the replica's own Content-MD5 check, and reads the stored bytes back.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* volume: fail the replica forwarding test instead of hanging when nothing is forwarded
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* volume: compare the primary and replica bytes in the replica forwarding test
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* filer.copy: compute chunk md5 so files get a real ETag
Chunks uploaded by filer.copy had no ETag, so every multi-chunk file got
the md5 of the empty string plus -N. An overwritten file kept its old ETag
and a revalidating client got 304 for changed content.
* filer.copy: store chunks in offset order
Chunks are collected in upload completion order, and the entry ETag
hashes the chunk ETags in stored order. Identical files copied with
concurrent chunk uploads got different ETags, and a file with swapped
equal-sized chunks could keep the old one.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* filer.copy: test chunk order and chunk md5 on chunked uploads
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* weed/shell: support -noLock flag in volumeServer.evacuate (#12501)
* shell: return flag parse errors from volumeServer.evacuate
A typo such as -noLock=invalid printed the parse error but the command
still succeeded, so a piped script treated the evacuation as done.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* shell: volumeServer.evacuate -h prints usage without failing
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* helm: set replication and disk on buckets created by the chart
* helm: test bucket replication and disk without ttl
* helm: reject unquoted bucket replication values
* helm: treat a null bucket replication as unset
A blank replication key in a values file failed the render with the
quoted-string error, while blank ttl and disk are skipped.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* helm: require a three-digit bucket replication
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* helm: note that removing bucket settings keeps the stored rule
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* helm: reject bucket replication above 255
fs.configure rejects an xyz value above 255, so the hook Job failed after
the release rendered.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* fix(shell): fs.log.purge purges every filer, not just one
Each filer keeps its own metadata logs in its own store, so purging a
single filer leaves the chunks referenced by the others and the volume
vacuum cannot reclaim anything.
The shell picked that single filer at random (commands.go:208) when no
-filer flag was given, and the admin script worker pinned it to
filers[0] (admin_script_handler.go:574), so the same command purged a
different filer on every run and never the whole cluster.
fs.log.purge now resolves the filer list from the master and purges each
of them, continuing when one is unreachable and reporting the failures
at the end. -filer=<addr>,... restricts the purge to a subset.
Also stops mutating the shared command environment per filer, so a
long-lived shell keeps its original filer.
* fix(shell): bound filer discovery and cover the cluster path
Address review feedback:
- resolveLogPurgeFilers waited on the master with context.Background(): a
shell whose master connection dropped would hang before purging
anything, where it used to purge the connected filer. Bound the master
round trip with a timeout and fall back to the connected filer when the
master cannot be reached or lists no filer.
- the tests only exercised the single-filer fallback. Rename that test to
what it actually covers and add one asserting every filer of the list is
attempted and the shared command environment keeps its original filer.
* shell: reject an fs.log.purge -filer list with no addresses
A comma or whitespace only -filer value fell through to master
discovery and purged every filer instead of the intended subset.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* shell: fall back to the shell's filer without a second master lookup
When the master listed no filer, FilerAddress() ran filer discovery again
and could wait out the shell's ConnectTimeout before failing. Also cover
master discovery with the fake discovery master.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* shell: test the fs.log.purge filer loop against a fake filer
The previous test looped over the addresses itself, so it passed no
matter what Do did.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Martin Morez <martin@morez.org>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* filer: validate bucket names before the SQL DDL hooks
The SQL store builds CREATE TABLE and DROP TABLE statements by formatting
the bucket name into the query. The DML paths already reject names that
fail VerifyS3BucketName, but the bucket-creation and bucket-deletion
hooks ran the DDL without that check, so a name reaching them through the
filer or WebDAV directory path (which skips the check for dot-prefixed
directories) could inject stacked statements. Apply the same validation
the DML paths use.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* filer: check that a valid bucket runs both DDL statements
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
* weed-db: end startup when password authentication fails
The failure branches returned the result of writing the error response,
which is nil on success, so the handshake carried on to ReadyForQuery.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* weed-db: test startup with wrong and unknown credentials
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* weed-db: check the auth error code and fail on stray read errors in the startup test
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* admin: escape file properties before rendering the modal
The Properties modal concatenated the entry name, path, MIME type and
extended attributes into HTML, so object metadata written over S3 could
inject markup into the admin page.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* admin: keep every property key when escaping
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* security: do not accept a path- or method-scoped filer token as admin
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* security: test that scoped filer tokens fail the admin token check
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* s3tables: let a table renamed across namespaces keep committing
* s3tables: keep the same-directory exception off a directory another table shares
A record written before metadata locations were confined to their namespace
(6f93f874) and claimed (#12473) can keep its files in another table's
directory, in another namespace. The exception that lets a table renamed
across namespaces keep committing to the directory it left behind accepted
such a record's commits too, where master refused them.
The namespace check now runs first, as on master. Only a location it refuses
is matched against the record's current directory, and the exception holds
only when no other record keeps its files there. Where other records' entries
sit is ignored, so a table created under the renamed table's old name, whose
files go to a directory of its own, does not take the exception away. The
scan runs only on that path, so commits inside the namespace cost what they
did.
Co-Authored-By: Claude <noreply@anthropic.com>
* s3tables: seed the table at the renamed table's old name
A rename may keep the old name taken, so creating a table there can be
refused; seeding the record keeps the test on the update path.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude <noreply@anthropic.com>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
filer: prefix bucket default collection, assign copy collection explicitly
S3 server-side copy paths (CopyObject/UploadPartCopy) assign volumes with an empty collection and relied on the filer's bucket fallback. The fallback used the raw bucket name, while PutObject uses getCollectionName(bucket) = <filerGroup>_<bucket>, so copies landed in a second, unprefixed collection.
Prefix the filer's bucket default with the filer group, and have the S3 copy path pass that collection explicitly, like PutObject. Direct filer writes keep path-rule precedence, so this only aligns S3 copy with S3 put.
Adds regression tests for the fallback, explicit collection precedence, and the S3 copy collection resolver.
* weed/s3api: avoid truncated marker rescan in ListObjects (#12540)
Signed-off-by: Tyagiquamar <mohdquamartyagi@gmail.com>
* fix(s3): validate marker directory candidates
Respect prefix filters and internal-directory exclusions when recovering marker candidates. Restrict lookups to direct children, propagate read failures other than not-found, and deduplicate directory marker echoes.
Add regression coverage and model promoted objects consistently in the XML-unsafe prefix fixture.
---------
Signed-off-by: Tyagiquamar <mohdquamartyagi@gmail.com>
Co-authored-by: Chris Lu <chris.lu@gmail.com>
* s3vectors: bound body buffering before authentication
* docs: include vector engine installation and source builds
* s3vectors: prevent MIME sniffing of JSON error responses
* docs: load vector Cargo configuration when building from source
* docs: note the native protoc requirement for Windows vector builds
* docs: make vector source checkout and working directory explicit
* ci: handle deb822 sources during Ubuntu mirror failover
* ci: bound Ubuntu mirror attempts and reject partial index updates
* vector: expose request metrics and registration-aware health endpoints
* vector: report immutable read cache and storage fetch metrics
* docs: describe vector engine health, metrics, and maintenance
* vector: count canceled requests when their handler is dropped
* vector: account for uncached large immutable storage reads
* vector: reliably release the stalled credential reader in tests
* vector: observe large reads as their streamed body is consumed
* vector: bound heartbeats on established master registration streams
* test: exercise hardened vector maintenance on Go and Rust volumes
* vector: register engines in their configured filer group
* vector: make retried index creation preserve compatible datasets
* vector: allow registration to advertise a separate address
* test: release stalled credential readers without blocking FIFO cleanup
* test: allow hardened vector coverage in a named filer group
* vector: discover engines in the same filer group as their gateway
* vector: fall back when the advertised address is empty
* admin: expose vector readiness, maintenance, and management tools (#12520)
* s3vectors: share vector discovery with trusted management clients
* admin: show vector engine readiness and recent maintenance activity
* admin: suggest S3 Vectors IAM actions from the supported operations
* admin: manage vector indexes, policies, tags, encryption, and query data
* admin: validate SSE-S3 support before creating encrypted vector buckets
* admin: discover vector engines in their registration group
* admin: probe vector engines concurrently with independent time budgets
* admin: verify encryption capability on the remote S3 gateway
* admin: defer discovery until after the HTTP server can start
* admin: distinguish unknown vector health services from failed engines
* admin: count workers that can execute vector index maintenance
* admin: allow read-only users to query vectors and inspect tags
* vector: scope gateway and admin discovery to the configured filer group
* admin: refresh policy state after mutations and serialize policy controls
* vector: drain active RPCs before closing replaced discovery connections
* vector: respect request cancellation while waiting for discovery
* vector: prefer another registered engine after a transport failure
* admin: check all masters before reporting no registered vector engines
* iam: honor attached vector policies without broadening legacy grants
* vector: allow short-lived discovery clients to close their connections
* vector: refresh cached master addresses for each discovery lookup
* iam: enforce vector bucket tag conditions in attached policies
* iam: authorize vector bucket creation with the requested tags
* test: exercise vector IAM grants and tag denies on Go and Rust volumes
* admin: reject authenticated vector mutations without an admin role
* test: run vector IAM assertions before the shared engine is stopped
* vector: add native mini startup, shell tools, and a runnable Compose example (#12522)
* s3vectors: share vector discovery with trusted management clients
* admin: show vector engine readiness and recent maintenance activity
* admin: suggest S3 Vectors IAM actions from the supported operations
* admin: manage vector indexes, policies, tags, encryption, and query data
* admin: validate SSE-S3 support before creating encrypted vector buckets
* admin: discover vector engines in their registration group
* admin: probe vector engines concurrently with independent time budgets
* admin: verify encryption capability on the remote S3 gateway
* admin: defer discovery until after the HTTP server can start
* admin: distinguish unknown vector health services from failed engines
* admin: count workers that can execute vector index maintenance
* admin: allow read-only users to query vectors and inspect tags
* vector: scope gateway and admin discovery to the configured filer group
* admin: refresh policy state after mutations and serialize policy controls
* vector: drain active RPCs before closing replaced discovery connections
* vector: respect request cancellation while waiting for discovery
* vector: prefer another registered engine after a transport failure
* admin: check all masters before reporting no registered vector engines
* iam: honor attached vector policies without broadening legacy grants
* vector: allow short-lived discovery clients to close their connections
* vector: refresh cached master addresses for each discovery lookup
* iam: enforce vector bucket tag conditions in attached policies
* iam: authorize vector bucket creation with the requested tags
* test: exercise vector IAM grants and tag denies on Go and Rust volumes
* admin: reject authenticated vector mutations without an admin role
* test: run vector IAM assertions before the shared engine is stopped
* mini: optionally start and supervise the vector engine
* mini: bootstrap vector buckets with existing Lance catalog helpers
* shell: manage vector buckets, indexes, and vector data
* docs: provide a runnable vector and maintenance worker Compose stack
* shell: scope vector discovery to the session and close its connections
* mini: honor the bind address for vector and metrics listeners
* mini: reserve vector ports before assigning gRPC and compatibility listeners
* mini: wait for vector storage setup and master registration
* mini: create requested vector buckets without requiring the Lance HTTP listener
* shell: refresh master addresses during vector discovery
* mini: resolve sibling vector executables with platform extensions
* vector: preserve uncertain creates and fix mini listener allocation (#12525)
* vector: retain catalog records when index creation has an unknown outcome
* mini: keep chosen listener ports reserved throughout allocation
* mini: resolve hostname binds before starting vector listeners
* vector: fix integration test imports and recover interrupted index creation (#12533)
* shell: register vector operations from the command package
* s3vectors: resume interrupted index initialization safely
* mini: exclude disabled WebDAV from listener allocation
* s3vectors: retain uncertain creates across refused retries
* s3vectors: claim pending indexes before retrying initialization
* s3vectors: compare resolved distance metrics during recovery
* s3vectors: preserve metadata keys when retrying initialization
* s3vectors: verify ownership before initializing new catalog records
* helm: deploy vector engines with TLS, probes, and monitoring
* helm: provision vector buckets through the S3 Vectors API
* helm: bracket IPv6 pod addresses for vector listeners
* helm: discover vector engines through the all-in-one master
* helm: accept vector component NetworkPolicy overrides
* helm: bind vector metrics to the pod address on both IP families
* helm: provision vector buckets through embedded and all-in-one gateways
* docs: use the full Helm name for external vector certificates
* helm: render vector instance labels as strings
* helm: allow vector bucket hooks to use restricted security contexts
* helm: honor vector security context enable switches
* helm: prevent duplicate vector monitoring labels
* helm: reject metrics flags with the default pre-metrics vector image
* test: skip vector chart compatibility checks when Helm is unavailable
* test: pass installer scripts through stdin on Windows
* ci: allow cold Intel macOS vector builds to finish
* ci: retain completed Rust builds when later platform checks fail
* ci: build, test, and publish vector engines for macOS and Windows
* installer: install native vector archives on macOS and Windows
* ci: run desktop vector checks for shared Rust and protobuf changes
* ci: publish vector releases from a separate job with write permissions
* ci: build tagged vector releases without restoring caches
* ci: propagate Windows bash path conversion failures
* grafana: add vector API, engine, cache, and maintenance panels
* helm: include vector panels with Kubernetes metric selectors
* grafana: preserve cluster and namespace identity when matching engine health
* grafana: retain engine identity when scrapes fail
* grafana: show zero cache hits before the first hit
* grafana: keep cluster and namespace identity in vector cache ratios
* topology: list collections without nesting the collection map read lock
Items() already takes the read lock and copies the values. Holding it
around the call made ListCollections take the read lock twice, so a
DeleteCollection waiting for the write lock in between blocked the
second read lock and wedged the master.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* topology: test ListCollections against a concurrent collection delete
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* master: move the KeepConnected stream cleanup into removeClusterNode
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* master: drop a filer from the lock ring only with its last registration
A filer's replacement can register under the same address before the
master ends the old stream. The cluster list counts both registrations,
but the old stream's cleanup removed the address from the lock ring
outright, leaving the live filer out of the ring until it re-registered.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* master: order a stream's cluster list and lock ring changes
The cleanup of an ending stream could see its address removed from the
cluster list, then a new registration of that address add it to the list
and the ring, and only then remove it from the ring.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* master: take a leaving filer's reconnect out of the lock ring
A filer that reconnects while leaving registers with LeaveLockRing set.
Its broken stream can still be open on the master, and since that
stream's end no longer drops an address that is still registered, the
reconnect removes the address from the ring itself.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* helm: say what the Lance worker needs from security.toml
Since #12470 the Lance namespace authenticates weed-worker by a token
derived from jwt.filer_signing.key, which the chart renders only with
securityConfig.jwtSigning.filerWrite. Without it, a gateway with
identities lists the worker no tables, and the lance_* jobs silently
find nothing to do.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* helm: warn that filerWrite also requires signed filer HTTP writes
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* s3: encrypt volume data when the filer is configured to
weed s3 read the filer's configuration at startup (buckets dir, filer
group, masters, metrics, maxMB) but not its -encryptVolumeData, so a
cluster that turned encryption on at the filer still had S3 uploads
stored plain, while uploads through the filer itself were encrypted.
The gateway now encrypts when either its own -encryptVolumeData or the
filer's is set; weed-vector, weed-worker and weed-vector-cli already
follow the filer's.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* vector: weed-vector, weed-worker and the CLI take volume encryption from the filer
They already encrypted chunks for the volume servers when the filer's
configuration said so, or when their own --encryptVolumeData did. With
weed s3 following the filer too, the filer's -encryptVolumeData is the
one switch for every writer, so the three flags are removed.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* S3 Vectors tests: turn volume encryption on at the filer
VECTOR_ENCRYPT_VOLUMES=1 set -encryptVolumeData on the gateway, the
engine and the worker. It now sets it on the filer alone, so the leg
checks that each of them follows the filer: the gateway's stored chunks
carry a cipher key (with a gateway that ignores the filer, the leg fails
on "the gateway stored a chunk without a cipher key"), and so do the
engine's.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* weed-worker README: volume encryption follows the filer
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* s3: help text says the filer's -encryptVolumeData turns S3 encryption on
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* S3 Vectors tests: plain legs check the engine stored no cipher key
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
vector: the engine's, CLI's and worker's help say what they do now
weed-vector's --masters and --filer-group said "reads through" and
--security-toml named only gRPC TLS, though the engine keeps tables on the
filers and volume servers and takes https.client, the read JWT key and the
SSE-S3 keys from it too; weed-vector-cli called itself a "query spike";
weed-worker's --filer-signing-key spoke of write credentials the
namespace no longer vends.
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
vector: weed-vector drops the ignored --namespace and --filer-signing-key
Since #12468 the gateway resolves every table it sends, so the engine has
no catalog to name or authenticate to; the two flags were kept only so an
older command line still started, with a warning. The direct-storage
engine is new, and release 4.52's command lines need changing anyway
(--storage-option is refused), so they are removed.
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
vector: refuse an SSE-S3 index when the engine has no key for it
The engine writes and reads an SSE-S3 bucket's files itself, so without
s3.sse.kek or s3.sse.key it failed inside Lance and the client saw only
InternalFailure. weed mini makes this easy to hit: it generates its key
for its own S3 server only. A table the gateway marks sse_s3 is now
refused up front with FailedPrecondition, which the gateway answers as
ValidationException with the reason.
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* s3: don't report a version delete as done when the filer lookup failed
deleteSpecificObjectVersion treated any getEntry error as "entry is gone":
- the null-version precheck returned success without deleting anything,
- the recheck after a failed rm returned success while the entry may
still be there,
- a failed .versions lookup was read as "not the latest version", so the
latest blob was removed without repointing the pointer first.
Only filer_pb.ErrNotFound now counts as a miss. Other errors fail the
delete, so the S3 client or the lifecycle worker retries it. The lookups
go through retryFilerOp, like the other .versions reads in this file, so
a single transient filer error doesn't turn into a 500.
* s3: include recheck error when a version delete fails
* s3: recheck a failed version delete once instead of retrying
---------
Co-authored-by: Mansur Akhmadi <10739460+mansur-akhmadi@users.noreply.github.com>
* volume.fsck: batch and deduplicate filer entry deletions
In volume.fsck with -findMissingChunksInFiler and -reallyDeleteFilerEntries, deleting orphaned filer entries was slow and CPU intensive due to tiny worker channel buffers (128), lack of batch processing, redundant HTTP DELETE requests for multi-chunk files, and lock contention on parent directory tracking.
Optimize by:
- Adding startPurgeBatchWorkers with 4096-buffered channels to prevent producer stalls and batch opportunistic deletes up to 64 items.
- Deduplicating missing chunks per file path during volume checks so that multi-chunk files are queued once.
- Processing batches with in-batch path deduplication and aggregating parent directories before updating purgedDirs in a single lock acquisition.
- Buffering temporary fid file writes (64KB) during the traversal to avoid millions of unbuffered syscalls.
* volume.fsck: report a failed write of the filer fid files
The fid files are written through a buffer now, so the last part of
each one reaches the disk only in the deferred Flush. Its error was
dropped: a full temp disk left a short fid file and the collection
still succeeded, and the volume check would then take the chunks
missing from it for orphans.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* volume.fsck: retry a batched path whose earlier delete failed
A path was marked done before its delete ran, so a second copy of it in
the same batch was skipped after a transient failure and the entry
stayed. Only skip paths already deleted.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* volume.fsck: keep reporting a failed DELETE response read
The non-verbose path now discards the body without checking the copy,
so a connection cut after the headers went unreported.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* volume.fsck: keep one purge worker starter and one delete function
startPurgeWorkers and the single-path httpDelete were left only as
wrappers the tests called. Give their names to the batch versions and
adapt the tests.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* s3tables: give a table created without a location its own directory
An Iceberg table created without a location kept its files under its name
path, so a rename left them where the next table of that name would land.
Such a table now gets s3://<bucket>/<namespace>/<name>-<uuid>, stored as the
record's warehouseLocation and returned by GetTable; its first commit writes
there without moving the root. Records without one keep resolving to their
name path, so existing tables need no rewrite. Lance datasets stay at their
name path, where Lance clients and the vector engine look for them.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* iceberg: create tables and views in a directory of their own
A table or view created without a location took its name path, or a
unique one only when that path was already taken. It now always gets
<name>-<uuid>, so a later rename never leaves its files under a name
another table can take.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* iceberg: load and commit a table at its warehouse location
A table created through S3 Tables without a location has no metadata until
its first commit; loading or committing it now uses the directory recorded
for it, and only a record without one falls back to its name path.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* iceberg: fail a stage-create whose marker is not saved
The marker is how the commit finds the location the stage-create handed
out. Without it the commit fell back to the name path while the client's
files sat under the returned location.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* test: expect tables created without a location in their own directory
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* test: rename a table that keeps its files elsewhere before reusing its name
Enterprise keeps the storage marker on a renamed table's files, so the name
of a table whose files sit under its name path stays taken there.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3tables: remove the table directory when a create backs out
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* iceberg: answer a create that lost the race with the registered table
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* iceberg: answer a view create that lost the race with the registered view
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* iceberg: fail a staged commit whose name another create registered
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* iceberg: remove the directory of a create whose name another table took
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* iceberg: keep the metadata of a staged commit whose name is taken
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* test: write the lakekeeper table's data under its warehouse location
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* iceberg: commit a staged table at the location of its own stage
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* iceberg: never continue another stage when a commit's own marker is gone
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* iceberg: test filer honors exclusive creates and token-guarded deletes
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* ec_balance: make the per-rack cap on global moves a worker setting
The global load-balancing phase planned at most 10 moves per rack per
detection run, hard-coded. On a cluster where one disk per machine holds
several times its siblings' shards, that is about 20 moves an hour against
tens of thousands needed, however many moves the worker can execute.
Add global_max_moves_per_rack to the plugin worker config. The default stays
10, so nothing changes unless it is raised. It is clamped to 1..5000 because
the planner reads 0 as unlimited and would plan a whole drain in one run.
Claude-Session: https://claude.ai/code/session_011UpvafcfUuC2mLxNAXLQJi
* ec_balance: clamp global_max_moves_per_rack before narrowing it to int
A value past the int32 range made ReadIntConfig return the default, so a
request asking for more than the maximum got 10 moves instead of 5000.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* ec_balance: test the configured global cap through Detection
The helper test still passed if Detection handed the planner a literal cap.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* ec_balance: plan no more global moves per rack than the detection keeps
Each global move scans the source node's volumes, and the detection keeps
at most max_results moves. With the cap at 5000, three skewed racks of
20,000 volumes planned 15,000 moves to keep 500, taking about 70 s of the
300 s detection budget; bounded by max_results it takes about 5 s.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* ec_balance: test the configured global cap with a result limit
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* filer.remote.sync: -allMounts writes back every mounted directory from one process
A filer with many remote mounts needs one filer.remote.sync process per
mount. -allMounts runs the existing -dir write-back once per directory in
the mount table, in one process: each with its own subscription, client id
(the filer keeps one subscription per client id), offset, processor and
retry loop. A supervisor reads the mount table every -mountsCheckInterval
(5s) and starts a write-back for each mounted directory that has none
running, 10 at a time, 500 ms apart. It never stops one because the live
table changed: a write-back ends itself where the -dir process exits, at the
first mount-table version it receives that ends its mount (#12472), and a
directory still mounted is started again from its saved offset, after a
pause that grows from 1s to 30s while it keeps ending within a minute. A
directory mounted while the process runs, without a saved offset, waits up
to 2 minutes for one (a mount followed by a pull sets it after the pull). A
write-back that ends at an unmount 5 times in a row while the table still
mounts it at the same location is logged as a warning.
To run under a supervisor, a follow:
- returns errMountGone at the end of its mount instead of exiting.
admitUntilMountEnds takes the end action: the -dir process keeps draining
and exiting as before; a -allMounts follow admits nothing more (events
after the end can still arrive in the same batch or before the stream
closes, including remote.unmount's purge and a remount), ends its stream,
drains, and saves no offset.
- drains its admitted jobs before it returns, however it ended, so a
successor never races a job still running for the same path.
- saves its offset only over the value it read at its start or last saved
(offsetGuard), both at its checkpoints and at its end. Otherwise a
write-back behind in the log can overwrite the offset remote.unmount
writes after its purge (or a mount after its mapping write), and its
successor replays the unmount and never writes the remounted directory
back. Once it sees the offset moved it saves nothing more and keeps no
resume position. A value whose write reported an error counts as its own
when read back.
- reads the stored offset once, before the mount table, and uses it as both
its start and its guard's value: read after the table, an unmount
completing in between would start a follow past the unmount's table.
- keeps a resume position for its next attempt only with the mount location
it was earned under and the offset stored at the time; it is dropped when
the directory is mounted elsewhere or someone else moved the offset.
- turns a panic in a job into that job's error, so one directory's failure
does not end every directory's write-back.
Fixes#12471
* filer.remote.sync -allMounts: retire a failed follow whose mount is gone
A follow that failed before its stream reached the unmount (or before it
started) was retried every 30s forever: DetectMountInfo reports the
directory not mounted, which is not errMountGone. No stream runs on the
retry path, so a directory the live table no longer maps is retired there
and started again by reconcile if it is mounted again.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* filer.remote.sync -allMounts: drop the restart pause of an unmounted directory
ended() left notBefore and ends behind for every directory ever mounted.
reconcile now drops them once the directory is neither mounted nor
followed.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* filer.remote.sync: log a -dir drain held at the mount end
The -dir exit drained with processor.Drain, so a remote call hung during
an unmount or move held the process silently; the follow's own drain
already warns every 5 minutes.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* filer.remote.sync: keep the resume position through a failed mount read
A transient DetectMountInfo error dropped the only position behind the
event a follow failed on. With -timeAgo and no checkpoint saved yet, the
retry then started at a later now minus timeAgo and skipped that event.
The next attempt that reads the table still checks the position against
the mount location and the stored offset.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* filer.remote.sync: judge a kept resume position by the directory's remote location
resumeFor compared the mount's raw location, so a -dir below a mount kept
its position when the same remote was mounted one level up (or down),
although its paths now map to other keys; a replay from that position
skips startAfterMountOffset and writes old events to the new keys. The
position now carries dirRemoteLocation and is checked against it.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* filer.remote.sync -allMounts: warn about a replayed unmount only from an unmoved offset
The warning counted every end at an unmount while the directory stayed
mounted at one location, so write-backs that each saved progress, or
unmount and remount cycles between two reconciles, also tripped it, and
it told the operator to set the offset to now, skipping events. An end
saves no offset, so a run of ends is one replay only while the stored
offset stays put: a moved offset now starts a new run.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* filer.remote.sync -allMounts tests: wait for the supervisor before moving the frozen clock
A reconcile or end recorded after the clock moved would start its wait
or pause at the moved time, which the frozen clock never passes.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* remote.mount: save the sync offset right after the mount mapping
A new mount had no saved offset, so filer.remote.sync -allMounts waited
the full allMountsOffsetWait before following it and then started from
the directory's creation time anyway. remote.mount and remote.mount.buckets
now save the time taken just before the mapping write, keeping an offset
remote.unmount saved for a remount.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* remote.mount: save the directory's creation time as the sync offset
The mount time skipped local files already in a -nonempty directory, which
a write-back with no offset replays from the directory's creation. Saving
that creation time keeps the start position and only ends the -allMounts
wait.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* remote.mount: warn instead of failing when the sync offset is not saved
The mount mapping is already saved by then, and a mount without an
offset still starts from the same creation time after the -allMounts
wait. Failing stopped remote.mount.buckets before the remaining buckets.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* ecbalancer: drain a full server through any target that can take a shard
The global phase paired the fullest server with the single emptiest one and
stopped for the whole rack when that pair had no legal move. Where every
volume already has a shard on each machine, no cross-machine move keeps the
spread, so on hosts running one volume server per disk the first server's
disk kept its pile while its siblings on the same machine stayed nearly
empty: the phase never planned a move.
Try sources fullest first and targets emptiest first and take the first pair
with a legal move, so such a server drains onto its own machine's emptier
servers. A same-machine move goes only to a server holding none of the
volume, so one disk never carries two of its shards. Per-machine counts are
indexed once per rack and pairs without a move are remembered until a move
can change them, so an uncapped drain stays fast.
Claude-Session: https://claude.ai/code/session_011UpvafcfUuC2mLxNAXLQJi
* ec_balance: keep global moves ahead of within-rack moves under the detection cap
Detection truncated the plan to the per-cycle job limit in planner order,
which puts the global moves last. A long within-rack backlog pushed all of
them past the limit, so even a planned global move never ran. Order dedup
and cross-rack first, then the capped global phase, then within-rack.
Claude-Session: https://claude.ai/code/session_011UpvafcfUuC2mLxNAXLQJi
* ec_balance: keep each volume's moves in plan order when promoting global moves
The admin scheduler runs only the first proposal per volume each cycle. A
global move promoted ahead of its volume's within-rack moves can take a shard
that one of those moves placed, so it ran while the shard was still on the
original server and failed.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* ecbalancer: order global targets by fullness after taking the shard
The search stops at the first target that would end up fuller than the
source, which only holds when targets are ordered by that same value. With
unequal capacities an empty one-slot server sorted first and ended the search
before a larger server that could take the shard.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* ecbalancer: free the source disk slot on a global move
The global phase cleared the moved shard's bits but left the source disk's
free slots and shard count unchanged, unlike every other phase. Use
releaseShard so a later pick of that server sees the freed disk.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* ec_balance: test that a capped detection keeps a global move
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* ecbalancer: test the same-machine move skips a sibling holding the volume
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* iceberg: keep a losing table create from deleting the winner's metadata
* iceberg: keep a staged file whose published record was renamed away
A create answers a conflict when its record was already visible and
another request changed it before the create could back it out. The
cleanup then looked only at the original name, so a record renamed in
between read as absent and the file it still points at was deleted.
S3 Tables now reports that case with a conflict of its own, and both
create paths keep the staged file on it instead of probing the name.
A conflict over a still-hidden record, lost to another create, still
removes the file.
* iceberg: discard a losing create's metadata only while it is still its own
The cleanup deleted the staged file by name. The path can be reused
before it runs: the table is dropped and purged, and another create
stages the same v1.metadata.json and registers it, which the losing
request then deleted.
Staged files now carry a per-request token in an internal extended
attribute, and the cleanup deletes with that token as the filer's
ExpectedExtended precondition, checked under the path lock the delete
runs under. A file that took the path since is left in place.
* iceberg: classify a typed S3 Tables error by its code, not its message
isS3TablesAlreadyExists matched "already exists" in the message before
looking at the code, so an InternalError whose text mentioned an
existing entry read as the name being taken: CreateTable answered 409
instead of 500, and registrationRefused discarded a staged file the
catalog may still point at. A typed error now decides by its code; only
an untyped one is matched by message.
* iceberg: pin stage-create on in the stage-create conflict test
Stage-create is on by default, but a shell that sets
ICEBERG_ENABLE_STAGE_CREATE=false turned the expected 409 into a 501.
Setting it in the test keeps the result independent of the environment.
* iceberg: test two creates overlapping at the canonical metadata name
The concurrent-creators test only lines the creators up before they
start, so whether two of them ever collide at v1.metadata.json is up to
the scheduler. The new test fixes the order: the second create runs
while the first has staged v1.metadata.json and not yet registered it.
It checks that every staging attempt is exclusive, that the second falls
back to v1-<uuid>, that both files hold their own metadata while both
are staged, and that the loser removes only its own file.
* s3tables: a create whose unmark write never went out stayed hidden
confirmChild counted any failure to take a create's mark off as possibly
published, the lookup that precedes the write included. That lookup
failing leaves the mark on, so the record stayed hidden; if another
create then adopted it, the back-out reported a published record that
changed, and the Iceberg catalog kept a staged file no record names.
The mark-removal calls now go through a wrapper that notes whether an
UpdateEntry went out without the filer refusing it. Only then may the
record have been seen. A refused compare, a missing entry, or a failure
before any write leaves the create's record hidden.
* s3tables: pass mayBeVisible into backOutChild
Fixes the build after rebasing onto the backOutChild split.
* volume: check a needle's CRC the first time a range of it is read
A ranged read never checked the CRC, so a replica with a damaged needle
served the damaged bytes with 206 and the reader never tried another
copy. The first ranged read of a needle now streams the whole needle
through the existing CRC check, with the usual pooled page buffers,
before writing any byte. A mismatch fails the read, so the reader falls
back to another replica. Needles that pass go into a bounded LRU, keyed
by volume, compaction revision, id, append time and checksum, so later
ranges of the same needle stream as before.
Fixes#5959 for ranged reads.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* volume: key verified ranges by volume load, share concurrent checks
From review: the cache key read v.SuperBlock.CompactionRevision without
the data file lock, racing a compaction commit that reloads the super
block, and a remounted copy of a volume inherited the old copy's
verified entries. Key entries by a load generation instead, an atomic
counter that changes on every load, including the reload in
CommitCompact, and cache a result only if no reload happened during
the check. Concurrent first ranges of a needle now share one check
through singleflight instead of each reading the whole needle.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* volume: keep hot needles verified, leave the io error streak to the range read
From review: Contains does not refresh an entry's place in the LRU, so
even a constantly read needle was evicted after 64k other needles were
verified; look it up with Get instead. The verification pass also
cleared the io error streak before the range read ran, so a range read
failing with EIO after a passing check never built up a streak; the
pass into io.Discard now leaves the streak to the range read, like the
meta read before a streamed GET.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* volume (Rust): check a needle's CRC the first time a range of it is read
Mirror of the Go change: a range served from the data file now checks the
whole needle's CRC before any byte of it is sent, so a damaged replica
answers 500 and the reader tries another copy. Needles that pass go into a
bounded LRU keyed by the volume's load generation, needle id, append time
and checksum; concurrent first ranges of a needle share one check, and a
check that passes leaves the io error streak to the range read.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* volume (Rust): a range over the whole needle checks its CRC
Go reads a range that covers the whole needle through the checked full-read
path. The Rust volume server read it from the data file unchecked, so
"bytes=0-" served a damaged needle with 206. Check the slice's CRC before
answering, as the full read does.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* volume: check the covered blocks of every range against CRCs recorded at the first check
The first ranged read of a needle now records the CRC of each block
(64 KiB to 1 MiB, at most 64 blocks below 64 MiB) while it checks the
whole needle. Later ranges read the block-aligned span around them and
write a block's bytes only once it matches, so damage that appears
after the first check is not served.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* volume (Rust): check the covered blocks of every range against CRCs recorded at the first check
verify_checksum now records a CRC per block (same sizes as Go's
rangeCheckBlockSize) and returns them from the cache; read_checked_range
reads the block-aligned span of a range and answers 500 if a block no
longer matches, so damage after the first check is not served.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
* sw-client: the filer and volume client library
The client the Enterprise repository's VFS and vector engine use, with
the same API: filer gRPC for entries, listings, volume locations,
assignments and remote caching; volume HTTP, plain or TLS, for needle
bytes; chunk resolution for sparse, overlapping and manifest chunks; a
location cache; and the write side. Enterprise-only protocol fields
(license, conditional mutations, hard links, KV listing) are left out.
Compiled against weed/pb/filer.proto like the other worker crates.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* weed-lance-worker: reads straight from the filers and volume servers
direct_read::DirectReads wraps the object store Lance opens so that
entries, listings and bytes come from the filers (gRPC) and the volume
servers (HTTP, signed with the read JWT), whatever the entry holds:
inline content, chunk manifests, sparse or overlapping chunks, the
filer's volume encryption, the gateway's SSE-S3 (sse.rs, with the KEK
from security.toml or the filer) and objects tiered to remote storage.
Writes still go to the inner store. The code and its tests are those of
the Enterprise repository's engine.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* weed-vector: read tables from the filers and volume servers
The engine asks the masters for the filers (and follows the list), takes
jwt.signing.read.key, grpc.client, https.client and the SSE-S3 keys from
security.toml, and opens every table with DirectReads innermost and the
read cache outside it. --masters is required: the gateway finds the
engine through them and the engine finds the filers. Writes still go to
the gateway.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* sse: chunks out of a manifest keep their own SSE-S3 keys
A chunk expanded from a manifest carries its id as fid, so keying the
per-chunk key cache by file_id gave every such chunk the first one's key
and IV. Keyed by the chunk's id however it is carried.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* direct reads: a silent filer is passed over like an unreachable one
Each filer attempt is bounded, so a filer that accepts the connection and
never answers does not hold the read or the startup lookup.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* sw-client: carry the assignment's fsync to the volume upload
The filer's storage rule may want a needle on disk before the volume
answers; the assignment says so and the upload now asks for it, as the
Go uploader does.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* weed-vector: argument tests pass the required masters; connect timeout for filers
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* direct reads: a failed read is tried once more against fresh locations
A fetch that fails forgets the volume locations it was made against; the
request now looks them up again and retries instead of failing on a
volume that moved.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* sw-client: an exclusive create that finds its entry, and needle deletion
create_entry marks ENTRY_ALREADY_EXISTS with EntryExists, so a caller can
tell a lost conditional create from a failure; the transport deletes a
needle whose entry was never created.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* sse: encode chunks as the gateway stores them
EntryEncoder writes SSE-S3 under a fresh object key wrapped by the KEK,
serialized with the IV, key id and commitment the gateway's reader
expects, each chunk under the IV at its offset; and the filer's volume
encryption under a per-chunk key.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* direct storage: writes, deletes and table removal without the gateway
A put uploads its chunks to the volume servers the filer assigns, at
the filer's chunk size, encoded as the bucket and cluster ask, and
creates the entry the gateway would: md5, mime, size, the ETag
attribute, the SSE-S3 attributes. A conditional create is the filer's
exclusive create, so Lance's manifest commit conflicts the same way;
the needles of a create that failed are discarded. A multipart upload
turns parts into chunks and derives its ETag from them. Deletes remove
the entry with its data, and remove_table a table directory. Copy and
rename, which Lance does not use on S3, are not offered.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* vector.proto: Table says whether its bucket encrypts with SSE-S3
The engine writes the table's files itself, so it needs what the S3 write
path knew: whether the bucket declares SSE-S3. storage_options go: the
engine reaches no S3 store.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* s3vectors: send the engine the bucket's SSE-S3 setting
The rule the S3 write path applies to a Lance bucket's files, factored out
of DeclaredBucketEncryption, decides it from the vector bucket's record.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* weed-vector: every table operation goes to the filers and volume servers
Opens, index creation and drops all go through the direct store, so the
engine has no S3 client of its own left to protect: --storage-option,
the endpoint pin and the request-option allowlist go, and lance's own S3
store is built unsigned and never dialed. The bucket's SSE-S3 setting
comes with the table; -encryptVolumeData encrypts chunks for the volume
servers as the gateway's flag does.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* s3vectors tests: the engine takes no credentials
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* s3vectors e2e: the engine writes what the gateway would have
Every leg checks that the entries the engine wrote carry the gateway's
ETag attribute, the hardened leg that their chunks are encrypted for the
volume servers, and a new test that an SSE-S3 bucket's files are written
under keys the gateway unwraps when it serves them over S3.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* cluster plumbing moves to the worker crate for the engine, worker and CLI
security.toml, the gRPC TLS connector and acceptor, the master dialer,
the filer discovery and the storage connection built from them now live
in weed-lance-worker (cluster, security, grpc_tls), where the worker and
the CLI can use them too. The server crate re-exports security and tls
under their old paths.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* weed-worker, weed-vector-cli: tables through the filers and volume servers
dataset::Storage replaces the S3 credentials every job and the CLI
carried: a table is opened and written through the cluster's storage,
encrypted as its bucket declares (read from the bucket's record on the
filer), and dropped there. Both take --masters and security.toml like the
engine; --access-key and --secret-key go, and with them the catalog's
x-api-key credential, which the cluster token replaces.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* lance catalog: the cluster token alone authenticates a cluster process
The engine, the maintenance worker and the CLI hold the cluster's filer
signing key and reach a table's files themselves. A request presenting
the cluster token and no other credential is theirs: it is authenticated
by the token, runs through the trusted manager, and is vended no S3
credentials.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* worker chart, README and test design follow the worker's new arguments
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* s3vectors tests: the worker and the CLI reach the cluster through the masters
The cluster gets a filer signing key, which the catalog authenticates
the worker and the CLI by; both take --masters in place of S3 keys.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* direct storage: encrypted chunks each start their own CTR stream
The gateway encrypts a multipart part from the IV derived for its offset,
from position zero; the encoder encrypted from the exact byte offset of
the object's stream instead, which agreed only at block boundaries. A
part of 3,000 bytes read back wrong; now tested.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* direct storage: keep what a lost reply saved, discard what a failed put left
A create the filer saved but did not answer, or an exclusive create
retried onto this very upload, keeps its chunks and counts as written;
the needles of a put that failed midway are discarded. The report prints
reads as reads.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* weed-worker: the security.toml it is given also holds its signing key
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* worker chart: the Lance worker mounts grpc.client's certificate
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* direct reads: the filer deadline bounds a listing page, not the listing
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* weed-vector: security.toml booleans read as viper reads them
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* weed-vector: a filer the master lists badly does not hide the others
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* weed-vector: register with the masters only once ready to serve
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* sse: every chunk after the first is written under a key of its own
Chunks of one object shared a key with IVs derived from their offsets, so
two chunks meeting inside a cipher block shared keystream. The first
chunk stays under the object's key and IV, which is how the gateway reads
an object of one chunk; every other chunk gets a fresh key and IV, as the
gateway's multipart parts have and as it reads an object of several.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* worker chart: the Lance worker is given the filer group
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* s3vectors e2e: the gateway serves engine-written SSE-S3 files of one chunk and of several
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* sw-client: the HTTP transport asks the volume to fsync when the assignment does
write_synced fell back to the trait default, which refuses fsync, so every
upload under a fsync storage rule failed before it was sent.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* sw-client: https.client.ca alone vouches for volume servers
With a CA configured, reqwest still trusted the system roots next to it;
weed processes trust only the configured pool.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* sw-client: an empty volume range is read without a request
A zero length underflowed the Range end, and a range past u64::MAX
overflowed it; the first now returns no bytes, the second an error.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* sw-client: a whole-needle answer to a ranged read is cut to the range
A server or proxy that ignores Range answers 200 with the whole needle;
the read returned its first bytes in place of the ones asked for.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* sw-client: a read size past u64::MAX reads to the end of the file
offset + size overflowed before the clamp to the file size.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* sw-client: an SSE-encrypted manifest is refused without a decoder
Only filer-encrypted manifests were; an SSE one went to the protobuf
decoder as ciphertext.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* sw-client: an empty write leaves the staged file size alone
A zero-byte write past the end grew the size to its offset, as a
zero-byte pwrite does not.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* sw-client: the chunk cache keeps no empty chunks
They add no bytes, so nothing evicted them and the map grew with every
distinct empty needle, even at capacity zero.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* sw-client: flushing with a zero chunk size is an error, not a panic
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* weed-vector: an empty master list fails at startup
"" split into one empty address that passed the check, and the engine
then waited two minutes for a filer before a generic error.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* lance cluster: an empty master list fails at startup
Cluster::new took "" as one master, so weed-vector, weed-worker and the
CLI waited two minutes for a filer before a generic error.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* direct storage: a failed create keeps its chunks when the filer cannot be asked
The lookup after a failed create is what tells a lost reply from a
refusal; when it fails too, the entry may be saved, so its chunks stay.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* weed-vector: a gateway-described table takes the shared unsigned store options
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* vector.proto: give sse_s3 a new field number and reserve storage_options
Field 4 held the storage_options map on master; a gateway from that build
would send a length-delimited value the engine cannot decode as a bool.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* weed-vector-cli: open s3:// URIs on the cluster's storage
A table named by its s3:// location skipped the store wrapper and opened
Lance's own S3 store, which has no credentials since the fallback flags
were dropped. It now opens like a namespace table and needs --masters;
local paths keep the plain builder.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* vector: an index id is plain path components, and drop trusts describe()
`resolved()` now refuses an id whose part is empty, `.`, `..` or holds a
`/`, since the colocated path would land elsewhere on the filer; and
`drop_index` no longer repeats the colocation check for the location
describe() already enforced for the configured schemes, which made a drop
on a memory:// table fail in the scheme test configuration.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* vector: a file delete takes only its entry; a table path is plain names
The store's delete of one file no longer asks the filer for a recursive
delete, so a path that names a directory does not take its content with
it, and `remove_table` refuses a location whose bucket or path component
is empty, `.`, `..` or holds a separator, which the filer would resolve
outside the table.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* vector: chunks are assigned to the bucket's collection
The filer assigns a chunk's volume from the collection the caller names;
the gateway names the bucket, prefixed by the filer group when the
cluster has one, so a plain assign landed the engine's chunks in the
default collection. The store now asks for `<bucket>` or
`<group>_<bucket>`, with the group taken from the cluster connection.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* vector: a put's content type becomes the entry's mime
Lance sets no content type today, so every entry was written as
application/octet-stream; a put that does carry one now creates the
entry with it, as the gateway does.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* vector: an upload whose create went unanswered is done with
A create the filer took and never answered (a timeout, or a transport
that reports the filer unavailable) may still be applied, so its chunks
are kept, as they already were for a create whose reply was lost. The
upload then records that it is done: a second completion is refused
instead of creating an empty entry, and an abort discards nothing.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* vector: an upload dropped unfinished discards its chunks
A request cancelled mid-write drops its multipart upload with neither
complete nor abort called, leaving its needles on the volume servers.
Dropping an upload that still holds chunks now discards them on the
runtime; a completed upload, or one whose completion is in doubt, holds
none.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* s3: a cluster process registers and deregisters in the vector namespace
The Lance catalog ran a cluster-token caller through the trusted manager
for every operation but the vector-namespace write check of register and
deregister, which still used the caller-checked manager and answered 403
for an index of a LANCE bucket. One `manager(r)` now picks the manager
for all three.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* vector: the worker's flags take two dashes
The worker parses its flags with clap and, unlike the engine, does not
accept Go-style single-dash long flags, so the harness's hardened worker
leg and the README's `-encryptVolumeData` did not start it. The README's
commands also gain the `--masters` the worker now requires.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* vector: the CLI takes --encryptVolumeData
An index build or optimize from the CLI wrote its chunks unencrypted for
the volume servers unless the filer itself asked for the cipher; the CLI
now takes the flag the engine and the worker have.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* vector: a bucket's encryption is read when a table is opened
The setting was cached for a minute, so a bucket recreated with SSE-S3
within that minute had its first tables written plain, and a record that
could not be decoded was read as "no encryption". A table open now reads
the bucket's record from the filer, once per open, and an unreadable
record is an error.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* vector: a removed table's files leave the object cache
`remove_table` asked the cache to forget everything under the table, but
the cache was built without invalidation closures, so the call was refused
and its error dropped: a data file read before the drop was still served
afterwards. The cache now supports the invalidation, and a refusal is an
error.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* vector: an abort keeps the chunks it could not discard
`discard_all` stopped at the first chunk a volume server would not give
up and dropped the rest from the upload's state, so a retried abort found
nothing to do and the needles stayed. Every chunk is now attempted, the
ones left are kept for the next abort or the drop, and the upload can no
longer be completed.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* vector: a chunk write that fails gives its needle back
When the volume server took the chunk but its answer was lost, the needle
stayed under a file id nothing would ever reference. The assignment is
known before the write, so a failed write now discards it, best effort,
before the error is returned.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* vector: the live worker tests require the cluster's masters
With WEED_LANCE_NAMESPACE set and WEED_MASTERS not, the tests seeded their
tables through lance's own S3 store, which reaches nothing, and failed
before a job ran. The missing variable is now named outright.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* direct storage: only the filer's missing-entry error means not found
Matching any "not found" text read failures such as an unresolved remote
storage as a missing entry, so a delete or a listing reported success over
data that was still there.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* direct storage test: compare the chunks' SSE-S3 IVs
Each chunk's metadata wraps its key under a fresh nonce, so the metadata
differed even if the key and IV were reused.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* direct storage: a filer passed over unanswered keeps the outcome open
When one filer timed out on a create and the next refused it, the refusal
came back without the no-answer mark, so the put discarded chunks the
first filer may still reference.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* lance test: start every concurrent writer before awaiting them
The lazy map spawned each writer only after the previous one finished, so
the commits never overlapped.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* direct storage: count failed uploads and deletes
The storage report counted failed reads and lookups but not a volume
upload, an object delete or a table removal that failed.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* S3 Vectors tests: run weed-vector-cli as the cluster's processes run
The CLI ran without the fixture's environment and outside its directory,
so under VECTOR_READ_JWT it could not read the volumes and under
VECTOR_TLS it found no security.toml.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* direct storage: a filer that fell away leaves the outcome open too
An unavailable error (connection reset, deadline, transport failure) can
arrive after the filer took the request, so a refusal from the next filer
no longer settles a create the first one may still apply: the chunks are
kept, as after a timeout.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* direct storage: a chunk write abandoned in flight gives its needle back
The needle is assigned before the write, so a request cancelled while the
write is in flight left it on the volume server with nothing to name it.
`Source::upload` is now `assign` and `write`, and the upload holds the
assignment across the write in a guard: a write that fails discards it
and one abandoned discards it on the runtime, while a write that returns
hands it to the entry.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* lance tests: sign for the namespace as the worker does
The live tests reached the namespace anonymously once the S3 keys went,
and an anonymous caller lists no tables, so detection proposed nothing.
They now take the cluster key from where the worker reads it
(WEED_JWT_FILER_SIGNING_KEY or security.toml) and log at info, so a table
a handler skips says why in the harness output.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* s3tables lifecycle test: run the Lance worker against the cluster
The worker leg of TestLanceTableLifecycle still handed the Rust tests S3
credentials and no masters. The mini cluster now has a cluster key, and
the tests get WEED_MASTERS and that key; passes with
WEED_LANCE_MAINTENANCE=worker (3 fragments became 1, 4 versions removed).
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* direct storage: name no collection outside a filer group, as the gateway
Without a filer group the gateway leaves the collection to the filer, so a
filer.conf rule for the bucket's path picks it; naming the bucket
overrode that rule, and the chunks landed where a bucket delete does not
look.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* direct storage: a read reports the entry's content type
A put now stores its ContentType as the entry's mime, but get_opts
returned no attributes, so the type did not come back on a read.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
* filer.remote.sync: judge the mount table before admitting later events
The unmount was decided inside a processor job, so events behind the
mapping write, such as remote.unmount's purge deletes, were admitted and
could delete remote objects before the exit landed. The stream callback
now reads each mount-table version before the next event, admits nothing
after the one that ends the mount, saves no offset, and drains the
admitted jobs before exiting.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* filer.remote.sync: ignore mount-table versions from before the mount
Without a saved offset the follow starts at the mounted directory's
creation second, so it replays mount-table versions written before
remote.mount added the directory. Each read as an unmount and the
process exited on every start. A version now ends the write-back only
when the version it replaced mapped the directory to its location.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* filer.remote.sync: judge the mount table by the mount's own key
-dir may lie below a mount, and DetectMountInfo resolves it to the
enclosing mount. Looking the table up by -dir found no mapping in any
version, so a replaced table never counted as mapped and the parent's
unmount was missed, admitting its purge deletes.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* filer.remote.sync: a table renamed away ends the mount
A rename of /etc/remote/mount.mapping to another name inside /etc/remote
was ignored, although the filer no longer has a mount table. The table is
now matched by its exact path on both sides of the event, so renaming it
away counts as removing it, and a file of that name elsewhere under
/etc/remote no longer counts as the table.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* filer.remote.sync: a mount on another remote is not the same mount
The mount-table check compared only bucket and path, so remounting the
directory on another remote storage at the same bucket and path kept
writing through the old client, and a replayed unmount from the other
remote stopped the write-back. The remote storage name now counts too.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* filer.remote.sync: check the drain at the exit itself
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* filer.remote.sync: resolve the mount covering -dir in each table version
Judging the table by the startup mount key alone missed a mount added at
-dir itself below that key: the key stayed mapped, so the nested mount and
its later removal were both admitted, and the removal's purge deletes ran
against the startup remote. Each version is now resolved the way
DetectMountInfo resolves the live table, and a different covering mount
ends the follow as a move.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* filer.remote.sync: an earlier mount's unmount still ends the follow
Skipping every version whose replaced table did not map the directory to
this mount also skipped the unmount of an earlier mount of the directory,
which a -timeAgo replay reaches, and admitted its purge deletes against
this mount's remote. Only a version in which no mount covers the
directory, replacing one in which none did, is now skipped as history
from before the mount.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* filer.remote.sync: map paths below -dir under its place in the mount
With -dir below a mount, every remote key was taken relative to -dir and
appended to the mount's location, so /mount/sub/x.txt uploaded to x.txt
instead of sub/x.txt and could overwrite another object. The processor
now gets the remote location of -dir itself; the mount's own location
still judges mount-table changes.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* filer.remote.sync: resume below a mount no earlier than the mount's offset
remote.unmount saves the offset of the directory it unmounts, after its
purge, and leaves the offsets of directories below it alone. A
write-back of -dir=/mount/sub resumed from its own offset after a
remount, replayed the unmount and exited, on every start. Without
-timeAgo it now starts no earlier than the mount's offset. Whatever that
skips was purged, or was synced by a write-back of the whole mount.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* s3tables: refuse a table or view location another record holds
A create, register, update or rename could name a location at, under or
above another catalog record's files, including the files a renamed table
left under its old name. The new record then loaded that table's metadata,
had credentials vended for its files, and could purge them on drop.
Each of those writes now scans the bucket's records and refuses a root
another record claims, and a drop of a record sharing its files is refused.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* iceberg: check a table's location before writing to it
REST create and stage-create write a metadata file, and stage-create
vends write credentials, before the catalog registers the table, so the
create gate now also validates the location and refuses one another
record holds. RegisterTable runs the same gate before it reads the
metadata file, instead of reading first.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* iceberg: keep a dropped location another record holds
DropTable and DropView purge the dropped record's location after the
catalog delete. A view recorded at a table's location took that table's
directory, record included. The purge now leaves a location another
record still holds.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3tables: authorize CreateTable before reporting the namespace
A caller without CreateTable got 404 for a missing namespace, 403 for an
existing one and 500 with decoder text for an unreadable one. The grant is
now evaluated against the bucket first, and only a caller that passes
learns the namespace is missing or unreadable.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* iceberg maintenance: skip a table whose files another record shares
Orphan removal and snapshot expiry delete what a table's metadata does not
reference. For two records over one location that deletes the other's
files, so detection leaves such tables alone.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3tables: count another record's catalog entry as claimed
A table keeping its files elsewhere still owns its entry directory, and a
view recorded there would take that entry with it on drop. The ownership
check now refuses a root at, above or under another record's entry too.
Maintenance still compares files only, so a table created over a renamed
table's old name does not stop that table being maintained.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3tables: check a claimed location again once the record is written
Two creates, or a create and an update moving a root, could both read the
same root free and both land. A create now checks again after writing its
still-marked record and backs out if another record claims the root, and an
update that moved its root checks again after the write and puts the previous
metadata back. Of two racing writers at least one sees the other.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* iceberg: enable stage-create in the location ownership test
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* iceberg maintenance: check for a shared location again before removing orphans
A job can run long after its table was scheduled, or come from a detector
that never checked, so remove_orphans reads the bucket's records itself and
leaves the files alone when another record shares the table's location.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3tables: check a renamed record's new root again once it is written
A record without a location moves its root with its name, and was checked
only before the rename wrote the destination, so a create landing on that
root in between could share it. The rename now checks again with the
destination written and backs out, like a create.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3tables: put a moved record back when its location cannot be checked again
An update that moved its root kept the move when the check after the write
failed to list the bucket, so a racing writer that relied on that check could
share the root. The move is now undone as it is when the root is taken.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3tables: hold a moved record's old root until the move lands
An update moving a record's root wrote the new metadata first and put the
old metadata back when the check that followed found the new root taken.
The old root was free in between, so a create racing for it could pass its
own check and then share it with the restored record.
The update now stores the metadata it is about to write beside the current
one, so the record holds both roots, checks the new root with that
reservation in place, and only then switches the metadata. A move that
backs out drops the reservation without touching the metadata, and an
update that keeps its root clears a reservation a crashed move left behind.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3tables: test a move racing another update of the record
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3tables: drop a move's reservation when a rename or delete takes the record
A rename or a decoupled delete that took a record while an update held its
move reservation left the reservation on the old name, where the move could
no longer clear it, and later creates of that name were refused as taken.
Both now clear it with the record's other attributes, and a failed rename
does not put it back.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* test: register a table from metadata under its own location
The Spark register test registered a live table's metadata file under a
second name, which now fails because both records would own the source's
files. It checks that refusal instead, and registers a copy uploaded under
the new table's own directory, with its location pointing there, to cover
reading and writing a registered table.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* sw-client: the filer and volume client library
The client the Enterprise repository's VFS and vector engine use, with
the same API: filer gRPC for entries, listings, volume locations,
assignments and remote caching; volume HTTP, plain or TLS, for needle
bytes; chunk resolution for sparse, overlapping and manifest chunks; a
location cache; and the write side. Enterprise-only protocol fields
(license, conditional mutations, hard links, KV listing) are left out.
Compiled against weed/pb/filer.proto like the other worker crates.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* weed-lance-worker: reads straight from the filers and volume servers
direct_read::DirectReads wraps the object store Lance opens so that
entries, listings and bytes come from the filers (gRPC) and the volume
servers (HTTP, signed with the read JWT), whatever the entry holds:
inline content, chunk manifests, sparse or overlapping chunks, the
filer's volume encryption, the gateway's SSE-S3 (sse.rs, with the KEK
from security.toml or the filer) and objects tiered to remote storage.
Writes still go to the inner store. The code and its tests are those of
the Enterprise repository's engine.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* weed-vector: read tables from the filers and volume servers
The engine asks the masters for the filers (and follows the list), takes
jwt.signing.read.key, grpc.client, https.client and the SSE-S3 keys from
security.toml, and opens every table with DirectReads innermost and the
read cache outside it. --masters is required: the gateway finds the
engine through them and the engine finds the filers. Writes still go to
the gateway.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* sse: chunks out of a manifest keep their own SSE-S3 keys
A chunk expanded from a manifest carries its id as fid, so keying the
per-chunk key cache by file_id gave every such chunk the first one's key
and IV. Keyed by the chunk's id however it is carried.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* direct reads: a silent filer is passed over like an unreachable one
Each filer attempt is bounded, so a filer that accepts the connection and
never answers does not hold the read or the startup lookup.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* sw-client: carry the assignment's fsync to the volume upload
The filer's storage rule may want a needle on disk before the volume
answers; the assignment says so and the upload now asks for it, as the
Go uploader does.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* weed-vector: argument tests pass the required masters; connect timeout for filers
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* direct reads: a failed read is tried once more against fresh locations
A fetch that fails forgets the volume locations it was made against; the
request now looks them up again and retries instead of failing on a
volume that moved.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* direct reads: the filer deadline bounds a listing page, not the listing
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* weed-vector: security.toml booleans read as viper reads them
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* weed-vector: a filer the master lists badly does not hide the others
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* weed-vector: register with the masters only once ready to serve
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* sw-client: the HTTP transport asks the volume to fsync when the assignment does
write_synced fell back to the trait default, which refuses fsync, so every
upload under a fsync storage rule failed before it was sent.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* sw-client: https.client.ca alone vouches for volume servers
With a CA configured, reqwest still trusted the system roots next to it;
weed processes trust only the configured pool.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* sw-client: an empty volume range is read without a request
A zero length underflowed the Range end, and a range past u64::MAX
overflowed it; the first now returns no bytes, the second an error.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* sw-client: a whole-needle answer to a ranged read is cut to the range
A server or proxy that ignores Range answers 200 with the whole needle;
the read returned its first bytes in place of the ones asked for.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* sw-client: a read size past u64::MAX reads to the end of the file
offset + size overflowed before the clamp to the file size.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* sw-client: an SSE-encrypted manifest is refused without a decoder
Only filer-encrypted manifests were; an SSE one went to the protobuf
decoder as ciphertext.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* sw-client: an empty write leaves the staged file size alone
A zero-byte write past the end grew the size to its offset, as a
zero-byte pwrite does not.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* sw-client: the chunk cache keeps no empty chunks
They add no bytes, so nothing evicted them and the map grew with every
distinct empty needle, even at capacity zero.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* sw-client: flushing with a zero chunk size is an error, not a panic
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* weed-vector: an empty master list fails at startup
"" split into one empty address that passed the check, and the engine
then waited two minutes for a filer before a generic error.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
* lance catalog: in-process DescribeTable for trusted callers
DescribeTable and DescribeTableInternal resolve a table's location and
storage options without the HTTP layer; the internal form runs the lookup
through a trusted manager so the caller's catalog grants do not gate a
resolution its own permission already authorized. Same code as the
Enterprise repository.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* vector.proto: Table carries its location and storage options
The gateway resolves the table it addresses, so the engine can open it
without a describe round trip of its own. Both fields are optional; an
empty location means the engine looks the table up as before.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* lance catalog: test the in-process describe
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* s3vectors: resolve the table before calling the engine
The gateway describes the table in process, through the trusted catalog
manager, and sends its location and storage options with every engine
call. Storage options are vended for the caller's principal, as the
Enterprise router does for queries. A catalog failure answers with the
code the engine would have given.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* weed-vector: open a table at the location the gateway sent
A request whose table carries a location is opened there, with the
storage options the gateway vended over the engine's fallback; the
catalog is described only for a table that comes without one, as from
an older gateway. dataset::open_at and dataset::drop_at take a
description the caller already holds.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* s3vectors e2e: run the engine without a catalog
VECTOR_ENGINE_NO_CATALOG=1 gives the engine an unreachable catalog URL;
every CI leg sets it, so the suite proves the gateway resolves each table
it sends.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* lance catalog: an internal describe may write a vector index
With credential vending on, the options vended to the engine followed the
caller's s3tables grants, which never allow writing the vector namespace,
so every write the gateway sent would have run with read-only credentials
and reads would have lost their version lease. The engine is that
namespace's writer and the caller's vector permission authorized the
operation, so an internal describe of a vector table vends write-capable
options; other namespaces keep the UpdateTable check.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* s3: resolve tables for the engine only while the catalog runs
With -port.lance=0 the gateway has no catalog to resolve through; the
engine then keeps looking tables up itself, as before.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* lance catalog: a deregistered table describes as NoSuchTable
So the gateway answers NotFoundException for it, not InternalFailure.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* weed-vector: trust a sent location only for a colocated vector index
A caller that reaches the engine could otherwise name any storage the
engine's credentials cover. A location is taken from the request only
when the id names a vector index and the location is its colocated path,
which the engine writes already; anything else is described through the
catalog as before.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* weed-vector: scope the read cache by storage endpoint
The cache was keyed by bucket and path alone, so the same path reached
through another endpoint would have shared entries. Each open wraps the
store with the cache scoped to the endpoint its options name.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* weed-vector: keep a configured endpoint over one a request names
A caller that reaches the engine could otherwise send its fallback
credentials to an endpoint of the caller's choosing. When the engine's
--storage-option names an aws_endpoint, a table whose request names
another one is described through the catalog instead.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* weed-vector: use request options only where credentials cannot leave
A request's storage options are taken only when they hold the keys a
gateway's describe produces, so a proxy or an endpoint alias cannot route
the engine's credentials elsewhere, and an endpoint only when it is the
configured one or the engine holds no credentials to send. Anything else
sends the table to the catalog.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* s3vectors e2e: give the engine its endpoint with its credentials
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* weed-vector: a drop checks the request's options like an open
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* weed-vector: an unpinned endpoint needs the request to sign for itself
Without configured credentials the S3 store still signs with the AWS
default chain, so an endpoint from a request is taken only when the
request skips signing or brings its own credentials.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* weed-vector: a trailing slash does not unpin the endpoint, and say so
A configured aws_endpoint written with a trailing slash matched nothing
the gateway sends, so every table fell back to the catalog in silence.
The comparison ignores the slash, and a fallback for a resolved table is
logged at debug.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* weed-vector: resolved by the gateway, or refused
The engine no longer has a catalog: --namespace and --filer-signing-key
are gone, a request whose table names no location is InvalidArgument, and
one whose location is not the index's own or whose storage options the
engine's credentials must not be used with is PermissionDenied. A drop
removes the colocated path it was given.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* s3: the catalog resolves vector tables whether or not its port is served
newLanceServer builds the Lance catalog for the S3 Vectors API; the
listener on -port.lance is optional.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* s3vectors tests: start the engine without a catalog
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* s3: the aux listener test builds the catalog before serving it
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* weed-vector: an unpinned endpoint takes only credentials the request signs with alone
Lance fills an option a request leaves out from the engine's environment,
under any of object_store's names for it (TOKEN, SESSION_TOKEN, ...), and
asks the AWS default chain when the key pair is incomplete. A request that
named its own endpoint with only an access key id, or a key pair while the
engine's environment held a session token, would have sent the engine's
credentials there. Such a request now needs a full key pair, and its own
session token whenever the environment would supply one.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* weed-vector: accept --namespace and --filer-signing-key, ignored
Both flags shipped in 4.52 and are documented, so dropping them made an
existing command line exit at startup with an unexpected-argument error.
They are kept hidden and ignored, with a warning.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3vectors design: the engine is started without a catalog
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* weed-vector: the trust check takes the environment it reads
The options test read the process environment through signs_alone, so a
host exporting AWS_SESSION_TOKEN or TOKEN failed it. The caller now
passes the variable names in; describe passes the engine's own.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* weed-vector: log both endpoints when a request's options are refused
The gateway turns the PermissionDenied into a bare AccessDenied and logs
nothing, so a pinned aws_endpoint written in another form than the one the
gateway sends (host case, a default port, an IP for a name) failed every
call with no trace of which value differed.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3: the Lance catalog sets the credential role whether or not Iceberg runs
-iceberg.credentialRole reached the S3 server only while the Iceberg
listener started, so with -port.iceberg=0 the Lance catalog installed its
vendor but vending stayed off, and the vector engine got no credentials for
the tables the gateway resolves. Both catalogs now set the role through
one helper.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3: set the credential role once, before either catalog serves
Both catalogs wrote the role as they were built; the second write landed
while the first was already serving requests that read it.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
---------
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
* admin: show disk usage on the dashboard instead of the data size
The dashboard's Total Size card summed the volume and EC shard sizes the
volume servers report. That leaves out deleted needles waiting for vacuum,
EC generations kept after a vacuum, and anything else on the disks, so on a
cluster with a lot of EC vacuum activity it showed far less than the disks
held.
The card now shows the bytes in use on the volume servers' disks, the same
statfs numbers the volume server list uses, with the disks' capacity, and
its trend line follows that number. Servers that report no statfs numbers
count their data size, as on the volume server list.
Claude-Session: https://claude.ai/code/session_011UpvafcfUuC2mLxNAXLQJi
* admin: compute the disk usage percentage in float64
current*100 overflows int64 past about 92 PB of used space, so the
dashboard card showed a negative percentage.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* admin: size the volume server disk bar with calculatePercent64
int(host.DiskUsage) truncates byte counts on 32-bit builds, so the bar
width was wrong for any disk over 2 GB there.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* admin: test the cluster disk totals across statfs and fallback servers
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* mount: keep retrying writeback chunk uploads through a volume outage
With -writebackCache, close() returns before the data is uploaded. When
the volume servers stay unreachable past the upload's own retry (~20 s),
the chunk was dropped and the file never appeared, with no error to the
application. Retry transient upload errors with the same bytes until they
succeed, unless the dirty pages were discarded. Permanent errors and the
synchronous mode keep failing as before.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* mount: test writeback upload retries and the cases that still fail
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* operation: keep the filer's read-only verdict on assign failures
UploadWithRetry flattened AssignVolumeResponse errors into a string, so
callers could not tell a read-only rule from an outage. Wrap the filer's
coded error so errors.Is(err, filer_pb.ErrReadOnly) works; the message is
unchanged.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* mount: retry writeback uploads the upload layer reassigns, never read-only
A filer proxy that cannot reach the volume answers 500 with an empty
body, which util.IsTransientError does not match, so that outage still
dropped the chunk. Also retry what operation.ShouldReassignUpload takes.
A read-only rule stays final even when its path or message contains a
transient word such as "unavailable".
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* mount: stop a writeback upload once its dirty pages are discarded
Destroy() was only seen after a failed attempt, so a truncate during the
retry wait still started another upload, and a chunk that finished after
its pages were replaced was attached to the new content. Check before
each attempt, and drop the chunk instead of attaching it; volume.fsck
reclaims it.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* mount: keep Destroy from slipping between the discard check and the append
truncateEntry destroys the pages and then installs the new entry, so a
Destroy landing after the check but before AddChunks put the old chunk
into the replacing entry. Hold one lock across the check and the append,
and take it in Destroy while setting the flag.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* mount: leave the Windows interrupt drain to the capped filesystem hook
Interrupt hooks run in reverse registration order, so the Windows hook ran
before the filesystem hook and waited on pending writeback flushes without a
bound. With writeback uploads now retrying through an outage, Ctrl-C could
wait until the volume servers came back. Unmount only, as on Linux, and let
the filesystem hook drain for at most 30 s and keep the write cache.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* mount: test a writeback upload whose assign keeps timing out
With every volume server gone the assign RPC times out until the filer
client gives up; that error must be retried like a refused upload.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* grafana: chart the remote-deletion tombstone gauges
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* grafana: describe the fetch hold that outlasts a full tombstone table
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* grafana: word the tombstone panel description like the gauge help
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* deps: bump seaweedfs/raft so the leader quorum check follows the election timeout
A single master resuming raft state self-elects on a 1 ms election
timeout and restores the configured one afterwards, but the leader kept
running its quorum check every 1 ms for the rest of the process. The
raft change re-arms that check from the current timeout.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* deps: bump seaweedfs/raft to the raft PR head with the extended test
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* deps: bump seaweedfs/raft for the non-positive election timeout guard
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* deps: bump seaweedfs/raft so a timeout change re-arms the leader quorum check
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* deps: pin seaweedfs/raft v1.2.4
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* filer: fold small chunks into manifests 256 at a time
Every metadata event carries the file's whole chunk list, so a file grown
by small appends re-sent each chunk record on every append until the
list reached ManifestBatch (10000). Chunks under 1 MiB now fold once 256
of them build up; full-size chunks still fold at ManifestBatch, so large
files keep their flat lists and read without a manifest fetch.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* filer: merge appended small chunks at the small-chunk fold count
The append path merges small chunks into full-size ones, but only at
1000 of them; the fold now hides small chunks in a manifest at 256, so
the merge would never see enough of them. Merging at the same count
keeps rewriting them into full chunks first.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* filer.remote.gateway: keep the remote object while a version under a delete marker is locked
Every version of a versioned bucket uploads to the object's key, so on a
remote without versioning that object is the only remote copy of the
latest version. A delete marker deleted it even when a version still held
a legal hold or unexpired retention. Keep it while any version of the key
is locked, the same rule the filer uses before pruning a remote entry.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* filer.remote.gateway, filer.remote.sync: say that earlier versions need remote versioning
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* filer.remote.gateway: delete a kept delete marker's object once the version lock is released
A delete marker kept for a locked version stays unstamped. When an
update or delete leaves a version that carried a legal hold or retention
without an active lock, look up the key's latest version and, if it is
still that unstamped marker, run the delete again. Only changes to
versions that carried a lock trigger it, so the noncurrent mark and tag
updates of unlocked versions do not send repeat remote deletes.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* filer.remote.gateway: test the kept delete marker retry through the bucket mount
The retry reads the key's versions through a filer client parameter, as
the rename helpers do, so a test can check the mapped object key and the
marker stamp, and that lock-releasing updates and deletes reach it.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* filer.remote.gateway: check the lock release lookup goes to the gateway's filer
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* filer.meta.backup: stream from the start of an unfinished walk
A walk only inserts entries, so when the first walk is interrupted the
next start walks again and streams from the second walk's start: deletes
and renames between the two walks never reach the backup store. Keep the
first walk's start time in the store until a walk completes and stream
from it.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* filer.meta.backup: start fresh once the stream has passed an earlier walk's start
A walk start left in the store by a crash between saving the offset and
clearing it, or by an interrupted -restart followed by a plain restart,
stops mattering once the stream saves an offset past it. Ignore it then
instead of replaying from it on the next walk.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* filer.meta.backup: fail the walk on an unreadable offset
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* filer: a full remote-deletion tombstone table keeps the lazy fetch off
At the cap a new delete under a remote mount was dropped with a log line
and isRemoteDeletionPending answered false for it, so the lazy remote fetch
resurrected deleted paths. While the table is full every path may be a
pending delete: isRemoteDeletionPending answers true until the table has
room again. The size and the full state are gauges
(filer_remote_deletion_tombstones, filer_remote_deletion_tombstones_full).
(cherry picked from commit f3be0ab965cde3bfd3053d3ac720fbbe3f68e98b)
* filer: a full tombstone table drains on reads and lifts after the unrecorded deletes age out
While the table is full, a lookup miss evicts expired tombstones and those
the write-back offset has passed, so the table drains without a mutation.
The full state lifts only once the table has room and the newest
unrecorded delete is past the TTL: an unrecorded delete has no event to
confirm it, so only the TTL makes it safe, as for a pending tombstone.
(cherry picked from commit 19394fe5d67798af7b0d755ac18c83c75e20a12b)
* filer: a full tombstone table drains per mount, only when its offset advances
Offset-based release applies to the tombstones under the mount whose
offset was read; another mount's tombstones answer to their own offset. A
read sweeps the table only once that offset advanced since the last drain
for the mount or the unrecorded deletes aged past the TTL, so a listing of
many missing children does not rescan the table per child.
(cherry picked from commit 9cf00e2e54d70e6ba530e6ec1b1ff4b502873847)
* filer: the tombstones-full gauge help names when it clears
---------
Co-authored-by: james <james@medable.com>
* filer: no lazy remote fetch inside a store transaction
A rename onto an existing target deletes the target inside the store
transaction, then CreateEntry's FindEntry misses the row and runs the lazy
remote fetch, which persists the archive copy on a second connection. The
store wrapper drops that call's deadline, so the UPDATE waits without bound
on the row lock of the transaction's own uncommitted DELETE while the
rename goroutine holds the path's exclusive locks: every later lookup,
rename and delete of the path on that filer waits forever.
A lookup whose context carries the store transaction skips the fetch.
(cherry picked from commit b69e9535c2597d0c8e856f031d3778fddfd4159c)
* filer: materialize remote-only rename paths before the transaction
With the lazy fetch off inside a store transaction, a source or target the
filer holds only on the remote would read as absent to AtomicRenameEntry
and StreamRenameEntry: a rename with FailIfTargetExists could overwrite it,
and a remote-only source would skip the deferred copy. PrefetchRemoteEntries
runs the fetch for both paths before BeginTransaction, under the path locks.
(cherry picked from commit e61f9718018669ef59768cd5eda25b6466d69597)
* filer: skip the lazy remote fetch only in CreateEntry's lookup inside a transaction
The guard on every lookup inside a store transaction hid a source or target
the filer holds only on the remote from the rename, and the prefetch before
the transaction did not reach the children of a directory rename. The
deadlock needs a write: CreateEntry's lookup after the transaction deleted
the path. That lookup alone skips the fetch; reads inside the transaction
keep it, and their persist on a second connection holds no lock the
transaction waits for.
(cherry picked from commit f555092c324780a8fe5314b238694551e3ef6be5)
* filer: mark a store transaction independently of the store's own key
ArangoDB and FoundationDB keep their transaction under other context
keys, so the CreateEntry guard keyed on "tx" missed them. The filer now
marks every context returned by BeginTransaction and the guard checks
that marker.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
(cherry picked from commit bd0505eb80)
* filer: keep the remote check for an exclusive create in a transaction
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
(cherry picked from commit b6aac15c3b)
* filer: test that a create outside a transaction still consults the remote
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
(cherry picked from commit 972daddd84)
---------
Co-authored-by: james <james@medable.com>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* s3api: mint a versioned PUT's id after the object write lock
putVersionedObject minted the version id before the upload and the lock
wait. A DELETE that committed while the PUT waited got a newer id, so a
conditional PUT whose If-None-Match passed against the delete marker was
stored behind it: accepted, but GET returned NoSuchKey.
The finalize now re-mints the id once the lock is held and the
precondition passed, so the version orders after everything the check
saw. An ambiguous routed attempt keeps its id, since recovery looks for
that entry.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3api: mint a routed versioned PUT's id after the upload
On the routed path the version id was still the one minted before the
upload, and RECOMPUTE_LATEST picks the newest name. A DELETE that
committed during a long upload left its marker latest over the accepted
PUT. Re-mint right before the routed transaction, as the lock path does.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* filer: stop unbounded metadata-log growth on append-only files
Two independent leaks make a filer metadata log grow far faster than the
data it describes. On a cluster whose live logs weigh a few hundred
megabytes, they produced tens of gigabytes of metadata per day.
1. Metadata events embed the whole old and new entry, chunk lists
included. An append-only file — a log or a journal written in small
pieces — therefore re-serializes every chunk it ever had on every
append, so the cost is quadratic in the number of chunks. A
2,700-chunk file emits a couple of megabytes per append. Cap the
carried chunk list at 256 entries and set
metadata_event_chunks_truncated so subscribers that need the content
read the entry from the filer instead of trusting the event.
Subscribers that only track names, attributes, or chunk deltas keep
working unchanged.
2. appendToFile uploads the chunk before committing the filer entry.
When the commit fails — store error, cancelled context, filer
shutting down — the chunk is stranded: no entry references it, so no
later delete can mark it, and neither vacuum nor fsck can see it
because both work from the entry index. The volume server keeps
counting those bytes as live data forever. Release the chunk when
the commit fails.
* filer: keep chunk lists in metadata events
Mount meta caches, peer filers replaying into their own store, and the
filer.sync, filer.backup and filer.meta.backup sinks all store the event's
entry as the file. Dropping the chunk list past a count left those copies
of a large file without content.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* filer: release a metadata log chunk the lookup or commit leaves unreferenced
The flush retries a failed append with a new upload, so a store outage
left one stranded chunk per retry. A failed lookup strands the chunk too.
Use DeleteUncommittedChunks, as the other write paths do: the chunk was
never part of the file, so path chunk-deletion rules do not apply.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* filer: keep a metadata log chunk the failed create stored
CreateEntry inserts a new entry before its parent folders, so a parent
failure returns an error with the entry already referencing the chunk.
Release it only when the store shows no entry, or an entry without it.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* filer: count a metadata log piece the failed create stored as appended
The flush retry would append the same bytes after it, and a piece cut
inside a record then breaks replay of that segment. The retry takes the
update path, which does not create parent folders either, so returning
the error only duplicated the piece.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Martin Morez <martin@morez.org>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* s3 lifecycle: send the listed entry's identity on walker deletes
The daily walker dispatched LifecycleDelete with no ExpectedIdentity, so
the server skipped its compare-and-swap and an object rewritten between
listing and dispatch was deleted. The list adapter now records each
listed entry's identity, computed the same way as on the event path, and
the walker sends it.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3 lifecycle: current-version expiration must target the latest version
The walker addresses the latest version of a versioned key by id. A
version file never changes, so its identity still matches after a newer
PUT, and the delete went ahead: a delete marker over the new version with
versioning enabled, or the new null version deleted while suspended.
LifecycleDelete now resolves the key's latest version for
EXPIRATION_DAYS/EXPIRATION_DATE requests that carry a version id and
answers NOOP_RESOLVED (VERSION_NOT_LATEST) when it is a different one.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3 lifecycle: check the latest version for unaddressed suspended expirations
With versioning suspended, an expiration without a version id reads the
plain entry, which is the null version but not necessarily the latest:
a newer real version can sit in .versions. The latest-version check now
runs in the expiration branch, by versioning state: for every suspended
request (an unaddressed one addresses the null version) and for enabled
requests that carry a version id.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3 lifecycle: test that each version sibling carries its own identity
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* storage: skip quarantined EC shards on local reads
A needle read tried the local shard first even when its disk was
quarantined after repeated IO errors, so every read paid the failing
disk's EIO latency before falling back. Treat such a shard as not local
so the read goes to peers or reconstruction, and read it only as a last
resort when no healthy copy answers. Scrub still reads it.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* volume: refuse EC shard reads from a quarantined disk
VolumeEcShardRead served peers from a shard whose disk was quarantined,
and it ignored the .ecx lookup error: a failed index read returns a
tombstone size, so the peer was told a live needle was deleted. Refuse
quarantined shards and fail the request on an index read error; a
needle missing from the index still reads, as before.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* storage: report quarantined EC shards deleted right away
Quarantining an EC volume only dropped it from later full EC heartbeats,
so the master kept routing reads to the faulty disk for up to 17
heartbeat intervals. The heartbeat that quarantines the volume now
carries its shards as deleted, once.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* storage: do not announce EC shards mounted onto a quarantined volume
MountEcShards and the shard load notifier reused a quarantined EcVolume
and announced the new shard, pointing the master back at the faulty disk
until the next full EC heartbeat. Skip the announcement, and fail the
mount so a caller moving a shard here keeps its source copy.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* seaweed-volume: count IO errors on local EC shard reads
Go's readLocalEcShardInterval feeds every local shard read into the EC
volume's IO error streak; the Rust needle read and recovery gather did
not, so a failing disk under local reads never moved toward quarantine.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* seaweed-volume: skip quarantined EC shards on local reads
Mirror the Go read path: a needle read and the recovery gather treat a
shard on a quarantined disk as not local, and read it only as a last
resort once peers and reconstruction fail. Scrub still reads it.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* seaweed-volume: refuse EC shard reads from a quarantined disk
Mirror Go's VolumeEcShardRead: answer Unavailable for a shard whose disk
is quarantined, so the peer tries the next holder or reconstructs.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* seaweed-volume: report quarantined EC shards deleted right away
Mirror Go: the volume heartbeat that quarantines an EC volume carries its
shards as deleted, once, instead of leaving the master to drop them at
the next full EC heartbeat.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* seaweed-volume: fail EC shard mounts onto a quarantined volume
Mirror Go's MountEcShards: the heartbeat already leaves such a shard
unannounced, so fail the mount and let a caller moving the shard here
keep its source copy.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* storage: keep reading EC needles when the shard lookup fails
A failed or short master lookup aborted the read before any shard was
tried, so the quarantined-disk last resort never ran once the master had
dropped those shards, and local shards went unread whenever the master
was unreachable. Read on with the cached locations, as the Rust server
already does.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* ec: log a failed last-resort read of a quarantined shard
When reconstruction and the quarantined read both fail, say why the local
read failed too. Go and Rust.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* ec: drop a shard whose mount was refused on a quarantined disk
The refused mount left the new shard loaded on the quarantined volume,
where it was neither served nor announced, and the in-memory claim drew
later copies of that shard back to the same disk. Unload it again,
keeping a shard that was already mounted there. Go and Rust.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* ec: finish a mount batch before reporting a failed shard
A shard that failed to mount, now including one refused on a quarantined
disk, returned before the sidecar reload, and in Rust before the
heartbeat notify, so shards mounted earlier in the same batch kept
checksum protection off and reached the master only at the next full EC
heartbeat. Stop at the failed shard but finish the rest first. Go and
Rust.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* ec: keep a peer's deleted answer when its stream fails after it
A shard read recorded the holder's deleted answer, and a stream error
after it failed the read: Rust discarded the flag, and Go let the next
source or the reconstruction result overwrite it. With the last-resort
read of a quarantined shard, that could serve a needle a peer had
reported deleted. Once the deleted answer is in, end the read on a later
stream error instead of failing it.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* seaweed-volume: use the platform media error in the quarantine heartbeat test
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* storage: rebuild EC intervals from quarantined shards when healthy ones fall short
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* seaweed-volume: rebuild EC intervals from quarantined shards when healthy ones fall short
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* volume: serve a peer's EC shard read from a healthy copy on another disk
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* seaweed-volume: serve a peer's EC shard read from a healthy copy on another disk
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* seaweed-volume: refuse a quarantined EC shard read instead of serving a sibling copy
A sibling disk's copy can have its own deletion journal, and an EC blob
delete is journaled on one local copy only, so the sibling could serve a
deleted needle.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* volume: refuse a quarantined EC shard read instead of serving a sibling copy
A sibling disk's copy can have its own deletion journal, and
VolumeEcBlobDelete journals a delete on one local copy only, so the
sibling could serve a deleted needle.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* seaweed-volume: check the disk an EC shard mounted on for quarantine
mount_ec_shard now returns the disk it mounted on, and the mount RPC
checks only that disk, like Go's MountEcShards. Before, a quarantined
sibling holding the same shard failed the mount while the healthy
disk's copy stayed mounted.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* storage: test an EC mount beside a quarantined sibling
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* seaweed-volume: check the quarantined-sibling mount test by its result
A server that remounts a shard on the disk already claiming it refuses
the mount there instead. The test now only requires that the result
matches whether the healthy disk ended up holding the shard.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* storage: check the quarantined-sibling mount test by its result
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* storage: expect the quarantined-sibling mount result from the disk the store picks
The test accepted a refused mount whenever the healthy disk stayed empty,
so a handler that refused before mounting passed too. Ask the store which
disk it picks first, then require success exactly when that is the healthy
disk.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* seaweed-volume: expect the quarantined-sibling mount result from the disk the store picks
Mirrors the Go test. add_location's cross-disk reconcile mounted shard 1 on
the healthy disk too, so the test now unmounts it there first, leaving the
quarantined sibling as the only claimant, as in Go.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* filer.remote.sync: bounded filer lookups, one chunk cache per process
The filer lookups on the sync path (currentEntry, isSuperseded,
uploadCurrentEntry, liveRemoteEntry, the mount directory read) ran with no
deadline. A lookup that never answers keeps its job alive, the job keeps
its path in the MetadataProcessor conflict index, AddSyncJob waits on the
condition and the daemon stops consuming events with no log line. Each
lookup now carries a 30 s deadline, so the job fails and the processor's
stop-and-resubscribe path runs.
remoteUploadReader built a chunk cache per upload; the cache's worker
goroutine is never stopped, so each upload leaked one goroutine and its
cache (21k goroutines, ~150 MB/h under lock-file churn, OOM at the memory
limit). One chunk cache per process.
(cherry picked from commit ff77fd222772d4f3b68f34d5ee39a66684f90e78)
* filer.remote.sync: a failed mount lookup at startup never resolves to now
collectLastSyncOffset returned time.Now() when the mounted directory lookup
failed, which skips every event since the saved offset. The saved offset is
used instead; without one the daemon exits so a restart retries.
(cherry picked from commit 8fd3124f5b3d3f8c3a291263a710047b4aff81ae)
* filer.remote.sync: no upload on an unanswered supersession lookup; bounded offset read
retriedWriteFile returns the supersession lookup's error instead of
treating an unanswered lookup as "not superseded": in a replay an older
snapshot could otherwise overwrite a newer remote object. The processor
retries the event. isSuperseded keeps its boolean form for the other call
sites. GetSyncOffset's KvGet carries a 30 s deadline so a startup that falls
back to the saved offset cannot hang on it.
TestGatewayUncachedRewriteIsStillUploaded asserted the upload on a failed
lookup; it now asserts the withheld upload and the lookup error.
(cherry picked from commit 8fc2e718f7cd8322615b4e40007be903ade2dbfa)
* filer.remote.sync test: keep the slices import the other gateway tests use
* filer.remote.sync: an unanswered lookup after a failed attempt ends the retry loop; an unread offset never resolves to now minus timeAgo
After a failed upload attempt the supersession lookup's error ended in
isSuperseded's boolean and the loop could upload the snapshot again
unverified. The lookup error now ends the loop (errLookupUnanswered, not
retried) and the processor retries the event. With -timeAgo set, a failed
read of the saved offset resolved to now minus timeAgo, skipping everything
older; the daemon exits instead so a restart retries.
(cherry picked from commit 55ca2a88e01d7b556d494a404b9cc263c27b2566)
* filer.remote.sync: an unresolved start offset returns to the retry loop instead of exiting
resolveLastSyncOffset returns the lookup or offset-read error; both callers
(directory and buckets) hand it to util.RetryUntil, which retries with
backoff instead of the process exiting.
* filer.remote.sync: keep doc comments on the functions they describe
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* filer.remote.sync: isSuperseded returns the lookup error itself
Its only production callers needed the error, so the boolean wrapper was
left to the tests. The tests now assert the error on a failed lookup.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* filer.remote.sync: bound the stamp written after each upload
updateLocalEntry's UpdateEntry was the last filer call on the sync path
without a deadline. A filer that never answers the stamp keeps the job and
its path in the conflict index the same way an unanswered lookup did. The
stamp carries IF_ENTRY_EQUAL, so a retried event after a timeout either
stamps again or skips a stamp that already landed.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* filer.remote.sync: an unread offset never resolves to the mount's creation time
With the offset read now bounded, a timed-out read fell through to the
directory's creation time and replayed every event already synced. The
error goes back to the startup retry loop, as with -timeAgo; a missing
offset still reads as zero without an error.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* filer.remote.sync: stamp a copy of the event entry
updateLocalEntry set the RemoteEntry and dropped the rename marker on the
caller's entry before the RPC. A transient stamp failure retries the same
event in place, which then read as already synced and skipped both the
upload and the stamp. The stub filer now applies a successful update the
way the filer does, since tests had relied on that shared object.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* filer.remote.sync: a directory not created yet starts now, as before
A gateway started before the first bucket looked up a missing buckets
directory, had no saved offset, and returned an error to the retry loop
on every attempt until a bucket appeared. A missing directory with no
saved offset has nothing to replay, so it starts now as it did before.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* filer.remote.sync: take the start time before the mount lookup
A bucket created between a not-found lookup and the end of the offset
read had its creation before a start taken afterwards, so the gateway
never created its remote bucket.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* filer.remote.sync: save the resume position when a subscription ends
A failed event ends the subscription with ErrResubscribe, and the retry
loop runs startup again, recomputing -timeAgo from a later now. The
periodic checkpoint only runs as events arrive, so a failure before the
first one could start the retry after the event it failed on. The resume
position is now saved whenever the subscription returns.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* remote_storage: bound the sync offset write like the read
The offset write runs in the subscription's checkpoint callback and, now,
when a subscription ends; a filer that never answers it stalled either.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* filer.remote.sync: say that any saved offset wins over the creation time
The creation time in seconds was compared with the offset in
milliseconds, which made any saved offset win. That is the behaviour to
keep: remote.unmount recreates the directory and then saves the offset,
usually in the same second, so a seconds comparison would start at the
creation second and replay the purge. The comparison now states it.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* filer.remote.sync: keep the resume position for the retry when its write fails
The retry in the same process now resumes from the position the ended
subscription returned, as filer.backup does, so a failed offset write
followed by an empty read cannot recompute -timeAgo past a failed event.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* filer.remote.sync test: give the same-second saved offset a fractional part
With whole seconds, returning the creation second resolved to the same
time and passed.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: james <james@medable.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* filer.remote.sync: bounded filer lookups, one chunk cache per process
The filer lookups on the sync path (currentEntry, isSuperseded,
uploadCurrentEntry, liveRemoteEntry, the mount directory read) ran with no
deadline. A lookup that never answers keeps its job alive, the job keeps
its path in the MetadataProcessor conflict index, AddSyncJob waits on the
condition and the daemon stops consuming events with no log line. Each
lookup now carries a 30 s deadline, so the job fails and the processor's
stop-and-resubscribe path runs.
remoteUploadReader built a chunk cache per upload; the cache's worker
goroutine is never stopped, so each upload leaked one goroutine and its
cache (21k goroutines, ~150 MB/h under lock-file churn, OOM at the memory
limit). One chunk cache per process.
(cherry picked from commit ff77fd222772d4f3b68f34d5ee39a66684f90e78)
* filer.remote.sync: a failed mount lookup at startup never resolves to now
collectLastSyncOffset returned time.Now() when the mounted directory lookup
failed, which skips every event since the saved offset. The saved offset is
used instead; without one the daemon exits so a restart retries.
(cherry picked from commit 8fd3124f5b3d3f8c3a291263a710047b4aff81ae)
* filer.remote.sync: no upload on an unanswered supersession lookup; bounded offset read
retriedWriteFile returns the supersession lookup's error instead of
treating an unanswered lookup as "not superseded": in a replay an older
snapshot could otherwise overwrite a newer remote object. The processor
retries the event. isSuperseded keeps its boolean form for the other call
sites. GetSyncOffset's KvGet carries a 30 s deadline so a startup that falls
back to the saved offset cannot hang on it.
TestGatewayUncachedRewriteIsStillUploaded asserted the upload on a failed
lookup; it now asserts the withheld upload and the lookup error.
(cherry picked from commit 8fc2e718f7cd8322615b4e40007be903ade2dbfa)
* filer.remote.sync test: keep the slices import the other gateway tests use
* filer.remote.sync: an unanswered lookup after a failed attempt ends the retry loop; an unread offset never resolves to now minus timeAgo
After a failed upload attempt the supersession lookup's error ended in
isSuperseded's boolean and the loop could upload the snapshot again
unverified. The lookup error now ends the loop (errLookupUnanswered, not
retried) and the processor retries the event. With -timeAgo set, a failed
read of the saved offset resolved to now minus timeAgo, skipping everything
older; the daemon exits instead so a restart retries.
(cherry picked from commit 55ca2a88e01d7b556d494a404b9cc263c27b2566)
* filer.remote.sync: an unresolved start offset returns to the retry loop instead of exiting
resolveLastSyncOffset returns the lookup or offset-read error; both callers
(directory and buckets) hand it to util.RetryUntil, which retries with
backoff instead of the process exiting.
* filer.remote.sync: keep doc comments on the functions they describe
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
(cherry picked from commit 952d80c4d3)
* filer.remote.sync: isSuperseded returns the lookup error itself
Its only production callers needed the error, so the boolean wrapper was
left to the tests. The tests now assert the error on a failed lookup.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
(cherry picked from commit 51f603b5ec)
* filer.remote.sync: bound the stamp written after each upload
updateLocalEntry's UpdateEntry was the last filer call on the sync path
without a deadline. A filer that never answers the stamp keeps the job and
its path in the conflict index the same way an unanswered lookup did. The
stamp carries IF_ENTRY_EQUAL, so a retried event after a timeout either
stamps again or skips a stamp that already landed.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
(cherry picked from commit c7338fb68f)
* filer.remote.sync: an unread offset never resolves to the mount's creation time
With the offset read now bounded, a timed-out read fell through to the
directory's creation time and replayed every event already synced. The
error goes back to the startup retry loop, as with -timeAgo; a missing
offset still reads as zero without an error.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
(cherry picked from commit 0a1b6c2944)
* filer.remote.sync: stamp a copy of the event entry
updateLocalEntry set the RemoteEntry and dropped the rename marker on the
caller's entry before the RPC. A transient stamp failure retries the same
event in place, which then read as already synced and skipped both the
upload and the stamp. The stub filer now applies a successful update the
way the filer does, since tests had relied on that shared object.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
(cherry picked from commit 1f332e5866)
* filer.remote.sync: a directory not created yet starts now, as before
A gateway started before the first bucket looked up a missing buckets
directory, had no saved offset, and returned an error to the retry loop
on every attempt until a bucket appeared. A missing directory with no
saved offset has nothing to replay, so it starts now as it did before.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
(cherry picked from commit c31497a9a3)
* filer.remote.sync: take the start time before the mount lookup
A bucket created between a not-found lookup and the end of the offset
read had its creation before a start taken afterwards, so the gateway
never created its remote bucket.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
(cherry picked from commit f535fca8f6)
* filer.remote.sync: save the resume position when a subscription ends
A failed event ends the subscription with ErrResubscribe, and the retry
loop runs startup again, recomputing -timeAgo from a later now. The
periodic checkpoint only runs as events arrive, so a failure before the
first one could start the retry after the event it failed on. The resume
position is now saved whenever the subscription returns.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
(cherry picked from commit 7be919ded1)
* remote_storage: bound the sync offset write like the read
The offset write runs in the subscription's checkpoint callback and, now,
when a subscription ends; a filer that never answers it stalled either.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
(cherry picked from commit f888ac6b8f)
* filer.remote.sync: say that any saved offset wins over the creation time
The creation time in seconds was compared with the offset in
milliseconds, which made any saved offset win. That is the behaviour to
keep: remote.unmount recreates the directory and then saves the offset,
usually in the same second, so a seconds comparison would start at the
creation second and replay the purge. The comparison now states it.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
(cherry picked from commit a0d60ee00f)
* filer.remote.sync: keep the resume position for the retry when its write fails
The retry in the same process now resumes from the position the ended
subscription returned, as filer.backup does, so a failed offset write
followed by an empty read cannot recompute -timeAgo past a failed event.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
(cherry picked from commit 39ef551bcf)
* filer.remote.sync test: give the same-second saved offset a fractional part
With whole seconds, returning the creation second resolved to the same
time and passed.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
(cherry picked from commit 4bb74382a6)
---------
Co-authored-by: james <james@medable.com>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* volume: a delete cancels the writes of its needle still in flight
A write queued behind the volume locks, for example behind another write's
fsync on a busy disk, is not in the needle map yet. A delete of that needle
in the meantime finds nothing, and the write lands afterwards, so the needle
comes back on this copy only while the other replicas have it deleted.
Each volume now tracks its writes from entry until they return. A delete
cancels the ones for its needle, and a cancelled write that landed is
tombstoned before it returns, unless the needle was deleted or rewritten
since.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* volume: a replicate delete that finds nothing cancels the pending write
Rolling back a failed chunk upload sends DELETE ?type=replicate to every
holder, including the slow one whose write timed out on the client. That
holder has not indexed the write yet, so the delete answered 404 and the
write landed live afterwards. Cancel the needle's pending writes, then look
again in case one finished in between. A plain delete that finds nothing
still answers 404 without cancelling: it is not sent on to the replicas.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* seaweed-volume: a delete cancels the writes of its needle still in flight
Mirrors the Go volume. Volume::track_pending_write registers a write until
its guard drops, delete_needle cancels the needle's pending writes, and
settle_pending_write tombstones a cancelled write that landed unless the
needle was deleted or rewritten since.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* seaweed-volume: track uploads and cancel them on a replicate delete
An upload is tracked from the handler, and a durable one from submission
to the write queue, where the guard travels with the request: a handler
dropped while the write waits in the queue does not untrack it, and the
worker settles it after the batch. A replicate delete that finds nothing
cancels the needle's pending writes and looks again, like the Go handler.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* volume: take a cancelled write back in the lock span that publishes it
The write tombstoned itself only after it returned, with the volume locks
released. In that gap a compaction could move the record so the offset
check skipped it, a close could drop it, and an identical write sent after
the delete could match the live record and report success before the
tombstone removed it. The check now runs right after the publish, under
the same dataFileSyncLock and dataFileAccessLock hold, in the plain, fsync
and batch write paths.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* volume: a delete cancels only the writes carrying its cookie
A delete with the right needle id but the wrong cookie could cancel an
in-flight write and get it tombstoned. Each pending write now records its
cookie, and a cancel only marks the matching ones.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* seaweed-volume: take a cancelled write back in the lock span that publishes it
Mirror the Go change: settle_pending_write ran after write_needles_grouped
returned, so a compaction, a close or an identical write later in the same
batch could slip in before the tombstone. drop_cancelled_write now runs
right after the publish, under the same sync and access lock hold, in the
synced run, the unsynced run and write_needle_unsynced_or_refused.
write_pending_needles carries each write's PendingWrite down to that
point.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* seaweed-volume: a delete cancels only the writes carrying its cookie
Mirror the Go change: a delete with the right needle id and the wrong
cookie no longer cancels an in-flight write. Each tracked write keeps its
cookie next to its cancel flag.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* seaweed-volume: refuse a queued write whose volume was remounted
The write queue tracks a write on the volume found at submit, but the
worker looks the volume up again by id. After a remount, a delete reaches
the new volume and cannot cancel the write registered on the old one, so
the worker now refuses such a write instead of landing it on the new
volume.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* seaweed-volume: decode the delete's type parameter like Go's FormValue
The delete handler matched the raw query text for type=replicate, so a
percent-encoded name or value was not treated as a replica delete and did
not cancel the write it undoes. Parse the query the way the upload handler
does.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* volume: test a replica delete landing in a write's fsync window
Drive DeleteHandler while a durable write is parked in its .dat fsync,
appended but not indexed: the replica delete answers 404 and the write is
taken back when it publishes, while a plain delete leaves it live.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* volume: flush a take-back tombstone before the index row that points at it
A durable write taken back during its fsync appends the tombstone after the
.dat sync, and the .idx sync that follows made its row durable while the
record itself could still be lost, which reloads the volume read only. The
fsync and batch paths now sync the .dat again first when a take-back
appended a tombstone; a failure stops the volume taking writes like a failed
index sync.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* seaweed-volume: flush a take-back tombstone before the index row that points at it
A durable write taken back in write_synced_run appends its tombstone after
the .dat sync, so the .idx flush that follows could make the row durable
while the tombstone record itself was still in the page cache. The run now
syncs the .dat again first when a take-back appended a tombstone; a failure
stops the volume taking writes like a failed index flush.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* volume: a skip-cookie batch delete cancels the writes of the file id it names
BatchDelete with SkipCookieCheck left the needle's cookie at zero, so the
delete's cancel matched none of the in-flight writes and a retry queued
behind it could land afterwards. The non-EC delete now carries the file
id's cookie, as the Rust volume server already does; the EC path keeps
cookie 0, which DeleteEcShardNeedle reads as "skip the check".
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* volume: release a held test write before the store closes
A failed wait ended the test before release ran, so the writer stayed parked
in its fsync and the store cleanup blocked on its lock until the package
timed out.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* s3tables: route REST RenameTable and refuse unrouted S3 Tables requests
PUT /tables/<arn>/<ns>/<name>/rename had no route, so it fell through to the
/{bucket} routers and ran as a PutObject into a bucket named "tables",
answering 2xx while the table kept its name.
- Answer any s3tables-signed request no S3 Tables route takes with 404
UnknownOperationException, before the health, domain and bucket routers.
- Route the AWS RenameTable REST call (newNamespaceName, newName, both
optional; versionToken).
- Honor versionToken on RenameTable, checked against the snapshot the claim
asserts so a commit between the two reads cannot slip a stale token through.
* s3tables: match the credential scope's service, not a region named s3tables
The unrouted-request fallback widens what a false match costs: an S3
client whose region is named s3tables had every request answered as an
unknown S3 Tables operation.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3tables: refuse an empty newName or newNamespaceName on REST rename
An empty field was read as omitted, so {"newNamespaceName":"other",
"newName":""} moved the table under its old name instead of failing
validation the way AWS does.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3tables: check the rename version token after the snapshot comparison
Once the separately read metadata is known to equal the claimed snapshot,
the token compares against the parsed metadata; no second decode of the
snapshot bytes is needed.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3tables test: cover policy and tag changes in the rename snapshot compare
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3tables test: give each REST rename subtest its own source table
The token subtests renamed the table the first subtest produced, so run
alone they failed on a missing table instead of testing the token.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* master: keep the hostname out of raft metric names
go-metrics prefixes gauge names with the hostname by default, which gives
every master its own metric family, e.g. SeaweedFS_<host>_raft_peers.
Prometheus already tells masters apart by the scrape target.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* master: send Hashicorp raft metrics through the go-metrics global raft uses
hashicorp/raft and raft-boltdb report through github.com/hashicorp/go-metrics,
but the master set up its prometheus sink on the armon/go-metrics global,
so a -raftHashicorp master exported no raft metric families. The sink is now
registered on the hashicorp/go-metrics global, and armon/go-metrics is no
longer a dependency.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* grafana: chart raft commit time and leader contact
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* grafana: keep the cluster in the raft panels
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* grafana: average raft commit time and leader contact over the rate window
The go-metrics summaries keep quantiles for 10s, so a p99 scraped every
30s misses most observations. The _sum/_count rates count every sample.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* grafana: leave raft panels empty when nothing was observed
Clamping the count rate plotted 0 ms for windows without commits.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* master: let the raft metrics test use its own registry
Registering on stats.Gather made go test -count=2 fail with a duplicate
collector registration.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* master: register raft metrics before opening the raft stores
raft.NewRaft starts its run loop before returning, so registering the
sink afterwards could drop the first state and boltdb samples.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* s3api: look up a trailing-slash prefix's directory instead of listing its parent
A prefix such as "account-0001/" names one directory, but the walk found it by
listing the parent with the name as a prefix. SQL stores answer that with a LIKE
that bounds the scan from below only, so in a wide bucket root the query reads
every sibling after the name: seconds for a prefix holding a few keys.
Look the directory up by name and serve it as the level's only entry. A miss,
an error or a non-directory falls back to the prefix listing.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3api: track failover receives atomically
The failover wrapper flags a received reply through a plain bool, which a
callback reading from the filer on several goroutines would race on. Use an
atomic.Bool so the callback may share its filer client.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3api: read directories ahead of a recursive listing walk
A recursive ListObjects walk enters directories one at a time, so a bucket
made of many directories holding one or two keys each costs one filer
round trip per directory, in series: seconds per 1000-key page on a store
whose reads wait on disk.
Read the first window of the directories the walk will enter on a few
workers, nearest first in walk order, and hand each window over when the
walk gets there. Every window the walk reads itself is drained first so its
subdirectories are queued before the walk descends into the first one. The
number of windows held ahead grows while directories end inside their first
window and falls back when one does not, so a page ending inside a large
directory leaves few reads wasted.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3api: drop the prefetcher's record of every directory it was handed
A page can cross any number of empty directories without spending its key
budget, so a set of every announced directory grows with the tree. A
directory is already handled when it is at or behind the walk, held or in
flight, or queued; check those, which the queue and window caps bound.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3api: end a trailing-slash prefix listing at the file its lookup finds
A file named by the prefix lists nothing under it, the same as the
sibling listing it fell back to, which threw the lookup's answer away
and paid the parent scan anyway.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3api: read a walk's own listing ahead instead of draining it
Draining the window before the walk took its first entry made a page
wait for the end of the listing, which a SQL store answering a name
prefix reaches only after scanning the rest of the directory; a page
that fills before then used to return at once. A background reader now
hands the window's directories to the prefetcher as they arrive while
the walk takes each entry, holding at most 1 MiB of entries ahead.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3api: bound the bytes a prefetched listing window holds
Entries with long chunk lists or inline content made the up to 64
held windows of 32 entries an unbounded amount of memory per listing.
A window now stops at 128 KiB of entries and the walk lists the rest
of that directory from the window's last entry.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3api: test recursive listings whose streams break partway
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* s3: require their own actions to change a bucket's CORS or versioning
PUT/DELETE ?cors and PUT ?versioning were authorized as Write, so an
identity granted only object writes on a bucket could replace its CORS
rules or suspend versioning. They now need PutBucketCors,
DeleteBucketCors and PutBucketVersioning (or Admin), and inline policies
granting s3:PutBucketCors, s3:DeleteBucketCors or s3:PutBucketVersioning
map to those actions instead of Write.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3: test that object Write cannot change bucket CORS or versioning
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* admin: offer the bucket CORS and versioning permissions
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3: test each bucket config action with its own identity
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* s3: commit the delete marker in the owner's finalize transaction
createDeleteMarker wrote the marker with mkFile to the gateway's current
filer and then sent RECOMPUTE_LATEST to the object's ring owner. When the
two differ, the owner scans .versions before peer metadata sync delivers
the marker, re-derives the pointer to the previous version, and the
marker stays non-current on every filer.
routedVersionedFinalize now takes an optional version entry and PUTs it
in the same ObjectTransaction ahead of the recompute, the way PutObject
and CompleteMultipartUpload already do. The copy destination keeps
passing nil: its version file is already written on the owner.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3: test a versioned delete when the gateway's filer is not the owner
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3: assert the delete marker and the pointer flip share one transaction
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* s3api: apply copy bucket-default SSE after the ACL re-auth
CopyObject without its own x-amz-server-side-encryption header takes the
destination bucket's default encryption, which CopyObjectHandler writes
into the request headers. It did so before preparePutObjectACL, which
re-authenticates a request that carries an ACL (x-amz-acl or a grant
header) for s3:PutObjectAcl. verifySignedHeadersCoverage then found the
synthesized x-amz-server-side-encryption* headers outside SignedHeaders
and rejected the copy with SignatureDoesNotMatch.
Move the bucket-default injection after the ACL check, so the
re-authentication sees the request as the client signed it. The
destination object still gets the bucket default encryption.
* s3api: assert the copy's KMS key and stored ciphertext
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Stefan Scharlott <stefans@Stefans-MacBook-Pro.local>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* admin: re-send the master vacuum disable every poll so it survives leader changes
The admin disabled the master's scheduled vacuum only when a vacuum plugin
worker connected or disconnected. The master keeps that flag in memory and it
is not replicated through raft, so after a leader change or a leader restart
the new leader ran its own vacuum next to the plugin worker. When both picked
the same volume the master's commit failed ("compact swap aborted: missing
.cpd/.cpx") and it dropped that replica, leaving the volume under-replicated.
Send the desired state on every poll instead, and log on the master only
when the flag actually changes.
* admin: shorten the vacuum sync comments
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Kevin <kevin@leetify.com>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* filer: hide write controls in the UI on the read-only port
The read-only port accepts only GET and HEAD, but its directory page
offered New Folder, Upload, drag-and-drop, rename and delete, each of
which failed with 405. Requests that come through the read-only handler
now render the page without those controls or the upload script.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* filer: test the directory page controls on each port
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* filer: match the rendered row buttons in the port test
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* weed/server: verify data shards against bitrot sidecar before ec decode (#12422)
* ec.decode: verify data shards against the mounted volume's bitrot protection
The decode looked up only the generation-0 .ecsum beside the data base and
each location's data directory, so it missed a sidecar kept with the index
and ignored the generation the volume actually serves.
Take the protection the volume's per-disk runtimes resolved instead (On, else
Invalid, else Off), and test a corrupt block past the first.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* ec.decode: check for live entries before verifying the shards
A volume whose needles are all deleted gets its shards purged on the
no-live-entries status; a failing checksum must not keep it around.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* seaweed-volume: verify data shards against the bitrot protection before a decode
Mirrors the Go volume server: the decode takes the protection the volume's
per-disk runtimes resolved and refuses a data shard that fails its .ecsum
checksum, after the no-live-entries check.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* test: a decode refuses a data shard that fails its .ecsum checksum
Runs against both volume server implementations: the refused decode keeps
every shard and the sidecar, and the needle stays readable through EC.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* ec.decode: reload the bitrot sidecar before verifying the shards
A rebuild's backfill writes the sidecar without reloading the mounted
runtimes, which then still report no protection; resolve it again across
the EC metadata directories, as a shard mount does.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* seaweed-volume: reload the bitrot sidecar before verifying the shards
Mirrors the Go volume server: the decode job resolves the sidecar again on
every runtime of the volume before choosing its protection.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* test: a decode finds a sidecar written after the shards were mounted
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* test: read through the EC shards after the refused decode
The source volume stayed mounted, so the final read never touched the
shards it is meant to check.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* ec.decode: refuse when a sidecar reload reports another layout
ReloadBitrotSidecar logged and dropped the layout-mismatch error, so the
runtime stayed Off and the decode skipped the data-shard check. It now
returns that error and the decode refuses on it.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* seaweed-volume: refuse a decode when a sidecar reload reports another layout
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* test: a decode refuses a late sidecar that records another layout
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* ec.decode: refuse when an unmount left no runtime to resolve the sidecar
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* seaweed-volume: refuse a decode when an unmount left no runtime to resolve the sidecar
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* weed/command: skip remote bucket delete when bucket is unmounted
* remote_storage: count deleting an already-missing bucket as done
filer.remote.gateway can see the same bucket delete more than once, for
example once per filer when several filers peer their metadata. A copy
that still finds the mount mapping calls DeleteBucket again, and S3
(NoSuchBucket), Azure (ContainerNotFound, ContainerBeingDeleted) and GCS
(404) all failed it, which held the sync offset on the delete. A
missing bucket now counts as deleted, the way deleting a missing object
already does.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* filer.remote.gateway: keep the mapping table writable after mount.mapping is deleted
Deleting mount.mapping replaced the gateway's mappings with a table whose
map was nil, so the next bucket create panicked writing its mount into
it. Reset to an empty map instead.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* filer.remote.gateway: count a delete of an absent object as done
filer.remote.sync already treats ErrRemoteObjectNotFound from DeleteFile
as deleted, since GCS reports a missing object that way where S3 and
Azure succeed. The gateway returned it, so a delete event seen twice, as
it is when several filers peer their metadata, held the sync offset on
GCS. Use the same deleteRemoteFile.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* filer.remote.gateway: drop cached remote clients after the new gateway tests
GetRemoteStorage caches clients by configuration, so a repeated run
(-count=2) got the first run's recorder back and saw no calls.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* filer.remote.gateway: keep the default storage across mount.mapping changes
The default storage for new buckets is derived at startup and kept only
in the gateway's copy of the mappings; mount.mapping never stores it.
Replacing or deleting that file dropped it, so a later bucket create
with no -createBucketAt skipped the remote bucket.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* admin: make the cluster lock's fairness window configurable
Scheduled jobs hold the cluster admin lock for their whole run. Once the
lock had been held for a minute, every new job waited until all running
jobs had released it, so with hour-long EC jobs dispatch ran in waves and
most workers sat idle between them.
-adminLock.fairnessWindow keeps the one-minute default; 0 lets new jobs
join the shared hold, and shell clients get the lock when the scheduler
next drains.
Claude-Session: https://claude.ai/code/session_011UpvafcfUuC2mLxNAXLQJi
* admin: wait for the blocked acquire before disabling the fairness window
The wake-up test slept 50 ms before SetFairnessWindow(0), so a late
acquire that started after the change joined without exercising the
Broadcast. Wait until it is registered as a waiter instead.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* weed/shell: resolve collection from topology when ec.decode is given only volumeId
When ec.decode is invoked with -volumeId for a volume that belongs to a named collection without explicitly specifying -collection, DoEcDecode was called with an empty collection string. Target volume servers then rejected VolumeEcShardsCopy with 'CopyFile not found ec volume id <vid>' because they looked under the default collection.
This commit introduces FindEcVolumeCollection to discover the collection name directly from topology metadata (checking EC shard info and regular volume info), ensuring DoEcDecode receives the accurate collection.
* ec.decode: resolve every batch volume's collection from the topology
The batch passed -collection through unless it looked like a regex, so a
comma list (a,b), a {n} quantifier, ALL or _default_ reached the volume
servers as a collection name and the copy found no shards. The selector
only picks the volumes; each one is decoded in its own collection.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* ec.decode: take the collection only from the volume's ec shards
The lookup returned the first record per disk, so a regular volume on an
earlier node answered before the ec shards on a later one. A decode needs
the shards anyway, so their records are the only ones to read.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* ec.decode: leave the -volumeId lookup to DoEcDecode
DoEcDecode already resolves an empty collection from the topology.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* filer.remote.sync: stop a touched file from being restamped forever
A metadata-only update that moves the mtime past RemoteMtime (touch, or
an O_TRUNC open of an empty lock file) goes through UpdateFileMetadata,
which changes nothing on the remote, and the entry is stamped with the
same RemoteEntry and a new LastLocalSyncTsNs. That stamp is an update
event of its own. shouldSendToRemote only looks at LastLocalSyncTsNs
when RemoteMtime and the mtime share a second, so it still reads the
entry as dirty, and the event is stamped again. Nothing is uploaded,
but the daemon keeps rewriting the entry as fast as it reads its own
events back.
Skip the restamp when the backend left the object alone and the event's
entry already carries a stamp taken after its mtime. The touch itself is
still stamped once.
* filer.remote.sync: log the skipped restamp
The other skips in processUpdateEvent log at V(2); this one did not, so
with -v=2 a touched file showed an "update meta" line with nothing after
it.
* filer.remote.sync: skip the restamp of a stamp's own event, whatever the clocks
The skip compared the entry's stamp with its mtime, so an mtime ahead of
the sync's clock (a skewed writer, touch -d) stayed past every stamp and
kept the loop going until the clock caught up. An event that changed only
the RemoteEntry is the sync's own stamp, which needs no other.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* filer.remote.sync: ignore the filer's commit timestamp when spotting a stamp
The filer gives every stored update a new CommitTsNs, so a stamp's event
never compared equal outside the RemoteEntry and a touched file kept being
restamped against a real filer. The test now assigns one per update.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* volume: bound replica forwards so a silent peer cannot hold a write
The Rust volume server forwards each replicated write to its peers with the
shared outgoing reqwest client, which had no connect or request timeout.
When a peer is silently partitioned (packets dropped, no reset) the forward
waits until TCP gives up, minutes later, and the write waits with it. A
chaos test (replication 001, one volume server partitioned) saw every write
on a volume with a copy there block.
- Replica forwards go through replica_request, bounded at
REPLICA_REQUEST_TIMEOUT = 10 s from connect to the response.
- Both outgoing clients start from outgoing_client_builder, which adds a
5 s connect timeout, so new connections to a peer that never answers
fail fast. The shared client gets no overall request timeout, since it
also serves lookups and other transfers.
Tests: a replica that receives the forward and never answers fails the
whole replicated write at the bound, naming that replica; a silent peer
fails the request builder at the bound on a paused clock; a connect that
stalls in the resolver fails at the 5 s connect bound.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* volume: give a replica forward more time for a larger body
The 10 s replica forward bound runs from connect to the end of the
response, so it also covers sending the body. A write near the 256 MB file
size limit needs about 26 MB/s to reach its replica in 10 s, so on a link
that is slower or busy the forward failed and the write returned 500 even
though the peer was answering.
A forward now gets REPLICA_REQUEST_TIMEOUT plus one second per MiB of
body. A bodiless forward keeps the 10 s bound (a 1 MiB write gets 11 s).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* volume: bound each Go replica forward like the Rust one
The Go volume server forwards a replicated write with a 10 s dial bound and
the transport's 30 s response header timeout. A replica that takes the
forward and never answers held the write for 30 s, and a body larger than
the socket buffers to a silently partitioned peer was held until TCP gave
up, since nothing bounds sending the body.
Each forward now runs under replicaRequestBound: 10 s plus one second per
MiB of body, the same bound the Rust volume server puts on its forwards.
Test: a replica that receives the forward and never answers fails the
write at the bound (300 ms in the test); without it the write fails at
30 s on the response header timeout.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* volume: keep proxy variables out of the stalled-connect test
reqwest reads HTTP_PROXY and ALL_PROXY by default, so with a proxy set the
test's request went to the proxy instead of the stalled resolver.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* volume: count a replica forward that hit its bound as a timeout
DistributedOperationResult joined the replica errors into a plain string,
and the uploader reports a cancelled request without wrapping the context
error, so a forward cut off at replicaRequestBound was counted under
server_error in the replication failure metric, as was a response header
timeout before it.
The joined error now unwraps to each replica's error, and a forward that
ran out of its bound reports context.DeadlineExceeded.
Test: the silent replica write raises the timeout failure count by one;
with the flattened join it stays put.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* volume: test that a large Go replica forward gets its body's bound
A replica answers after the bodiless bound but within the bound for a
1 MiB body, so the write fails if replicateTo drops the body scaling.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* volume: run the Rust whole-path replica test with a 1 MiB body
The forward then has to wait the bound for its body, so the test fails if
do_replicated_request stops passing the body length to the bound.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: pingqiu <pingqiu@gmail.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* s3api: name the check that a completion already committed
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3api: answer success when a failed completion already committed
A finalize can fail after its commit: the filer stored the object and the
call still reported an error. The client then got a 500 with an unknown
outcome. When the failure may have left this attempt's object behind,
check the key the way a retry would and answer success with the
committed result when it holds this upload's object.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3api: keep part data when the commit found is another attempt's
The object carrying this upload ID can be a concurrent attempt's commit of
the same upload, and its part selection can include entries this attempt
counts as unused. Free those parts' data only when the committed chunks
are this attempt's own; otherwise drop their metadata only.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3api: test the committed-completion read on versioned buckets
Cover the latest-version read (version id and checksum in the answer),
a suspended bucket whose delete marker is still current, and that the
upload directory is removed without its data.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3api: test versioned recovery through the completion handler
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* seaweed-volume: bound each FetchAndWriteNeedle replica write
A replica that accepted the connection and then went silent held the
request until TCP gave up, because the shared outgoing client has no
request timeout. Each replica write now fails after 30 s, from connect
to the response status, matching the Go volume server.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* volume: bound each FetchAndWriteNeedle replica write
The default replica path uploads through the guarded peer client, which
bounds only the dial and has no response header timeout, and the filer
calls FetchAndWriteNeedle with no deadline. A replica that accepted the
connection and went silent held the request for minutes. Each replica
write, retries included, now has 30 s.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* seaweed-volume: require the silent-peer test to fail at the deadline
A connection error that came back early also passed, since the test
only checked an upper bound. It now also requires the paused clock to
have reached FETCH_AND_WRITE_REPLICA_TIMEOUT.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* seaweed-volume: grow the FetchAndWriteNeedle replica bound with the chunk
The reqwest timeout includes sending the body, and a remote-cache chunk
is a thousandth of the object once it passes the 1000-chunk cap, so a
fixed 30 s would fail a large chunk on a link under ~35 MB/s. The bound
is now 30 s plus 1 s per MiB.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* volume: grow the FetchAndWriteNeedle replica bound with the chunk
The context deadline includes sending the body, so a fixed 30 s would
fail a large remote-cache chunk on a slow link. The bound is now 30 s
plus 1 s per MiB, matching the Rust volume server.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* volume: redial a master that just dropped a registered stream after a second
A registered volume server that loses its master with no peer naming a
leader waited a full pulse before redialing. A restarted master is
usually back within a second, so its volumes stayed unregistered, and
uncached lookups kept failing, for most of a pulse.
For one pulse after a stream a leader had answered ends, redial on the
one-second cap. A master still down after that is redialed at the pulse
pace, so redials never come later than before.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* seaweed-volume: redial a master that just dropped a registered stream after a second
Mirror of the Go heartbeat change. The loop keeps when the last stream a
leader had answered ended, which also stands in for the
confirmed-leader-once flag (None until first registration), and redials
on the one-second cap for a pulse after that.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* seaweed-volume: test that a dropped registered stream records the loss
A fake master confirms the stream and then ends it. The server starts as
registered long ago, so only a recorded loss makes the redial come after
a second instead of a pulse.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* volume: record a lost stream only after its receiver has exited
doHeartbeatWithRetry can return on a send error while the receiver is
still handling the leader's first reply, so the deferred recorder could
read the registered flag too early and keep an old loss time. Cancel the
stream and wait for the receiver before reading it.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* test: require the prompt heartbeat redial to wait out its second
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* seaweed-volume: fail the down master's dials at once in the redial test
Windows takes about two seconds to refuse a connection to a closed
loopback port, so the stream dial and the leader lookup each held the
prompt case past its two-second bound. A listener that closes every
connection fails both dials immediately on every platform.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* ec.rebuild: rebuild shards that fail a checksum scrub, also in complete volumes
ec.rebuild skipped every volume with all shards present, so a shard whose
bytes no longer match the .ecsum sidecar stayed in place until some other
shard went missing. With -checksum, ec.rebuild first scrubs each node's
shards in checksum mode and takes the failing copies out of the plan; a
shard left without a good copy counts as missing and is regenerated.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* ec.rebuild: remove the copies that failed the checksum once a good copy serves
A shard regenerated on the rebuilder left the failing copy mounted on its
old holder, so the volume kept a corrupt duplicate that a later dedup could
keep instead of the good one. Remove each failing copy once the rebuilt
shard is mounted, and right away when another holder already serves a copy
that passed.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* ec.rebuild: test the checksum scrub against fake volume servers
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* ec.rebuild: check the lock before removing a corrupt copy
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* ec.rebuild: print the checksum scrub details
A malformed sidecar or a wholesale mismatch comes back only in the scrub
details, with no broken shard, so the volume would look verified.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* ec.rebuild: fail the run when a corrupt duplicate cannot be removed
For a volume whose only problem is a corrupt duplicate, the removal is the
whole repair, so its failure has to reach the caller.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* ec.rebuild: test duplicate removal failures and a lost lock
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* ec.rebuild: fail the run when a node's checksum scrub fails
The node's copies stay in the plan unverified, so a corrupt shard there is
left in place; the rest of the cluster is still repaired.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* ec.rebuild: preview the corrupt-copy removals in dry-run
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* ec.rebuild: test a failed checksum scrub
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* seaweed-volume: stage optional EC sidecar copies before replacing
VolumeEcShardsCopy opened the destination .vif/.ecsum with File::create
and then streamed into it. A source without the file left a 0-byte .vif,
wiped an existing .ecsum (turning bitrot protection off on that holder),
and a failed stream removed the destination's existing copy.
Stream optional copies into <file>.copying and rename into place only
once the source sent bytes, as the Go volume server's writeToFile does
for ignore_source_file_not_found.
Claude-Session: https://claude.ai/code/session_011UpvafcfUuC2mLxNAXLQJi
* seaweed-volume: keep the drain doc comment on drain_copy_stream_to_file
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* seaweed-volume: stage the .vif copied during missing-index recovery
fetch_ec_index_from_one_peer drained the optional .vif straight into the
local file, so a peer without one left a 0-byte .vif (losing a local EC
ratio) and a failed stream left a partial one. Stage it and rename only once
the peer sent bytes, as Go's recovery does through writeToFile.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* seaweed-volume: test optional copies against a stream that fails mid-transfer
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* seaweed-volume: remove a staged optional copy when the RPC is cancelled
A cancelled CopyFile handler drops its future at the stream await, so the
cleanup after it never ran and the .copying file stayed behind. Go's
writeToFile runs to completion and removes it. A guard now owns the
staging cleanup and is disarmed only once the rename commits.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* seaweed-volume: test optional copies dropped mid-stream
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* seaweed-volume: cancel the staged-copy test only once the staging file exists
tokio's File::create runs on the blocking pool, so a fixed timeout could
drop the .ecsum copy before .copying existed and pass without exercising
the cleanup guard.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* seaweed-volume: run the preallocating max-volume test on a fake disk
The test sized the volume limit from the real free space but the store
subtracts the 1% floor first, so on a disk more than 95% full the second
slot no longer fit and the count came out 1.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* seaweed-volume: start the preallocating max-volume test above the expected count
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
A query's lease tag is deleted by a background task after the query
answers, so the encryption helper could list it and then get a 404 on
HEAD. Skip only that case; any other missing object still fails.
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* seaweed-vector: build and extend the vector index
weed_lance_worker::vector_index builds an IVF_FLAT index in the metric the
table was created with, sizes partitions from the row count, and extends an
index with a new segment or a retrain; the core crate re-exports it and adds
the probes a query spends. Tests cover coverage after build, append, retrain
and compaction.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* weed-vector-cli: index build and index optimize
Logs go to stderr so stdout stays the JSON a caller parses.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* test/s3vectors: PERF_INDEX builds the index before the timed queries
The cluster fixture hands out its catalog URL, the Makefile builds
weed-vector-cli, and the perf suite builds the index after ingest and
reports the build time and segment count.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* seaweed-vector: probe a tenth of the index's partitions, more under a filter
A query that names no probe count gets one from the index's partition
count. Lance searches at least that many partitions, and under a prefilter
keeps going until it has k rows; without one it stops there.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* weed-vector: --nprobes overrides the probe policy
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* seaweed-vector: test that a build takes the table's metric
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* test/s3vectors: make clean removes weed-vector-cli
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* seaweed-vector: measure recall with vectors the table does not hold
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* weed-vector: keep the read cache's doc comment on with_read_cache
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* weed-worker: lance_vector_index builds the index of an S3 Vectors table past a row threshold
Detection keeps the tables a vector bucket created, with at least min_rows
rows (50,000 by default) and no index on their vector column. Execution
builds the IVF_FLAT index in the table's metric with sqrt(rows) partitions.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* seaweed-vector: optimize only the indices on the vector column
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* seaweed-vector: resolve the query metric with index::distance_metric
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* seaweed-vector: test the L2 metric under the name CreateIndex stores
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* test/s3vectors: build the index through the admin server and the Lance worker
VECTOR_WORKER=1 adds weed admin and weed-worker to the cluster. The new
test sets the job's threshold, runs it through the plugin API, checks
coverage and recall, and that a row written after the build is found.
CI gains the worker leg for both volume servers.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* weed-worker: read an index's partitions, and when a table has outgrown them
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* seaweed-vector: probes from the shared partition reader, partitions in coverage
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* weed-worker: a vector index reindexes at a tenth of its rows and retrains once the table outgrows it
lance_optimize_indices proposes a vector table once its uncovered rows pass
min(the absolute budget, a tenth of the indexed rows), and trains the index
again instead of extending it when sqrt(rows) has reached twice its
partitions. The end-to-end test covers both.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* weed-worker: open a table inside a caller's Lance session
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* weed-vector: keep opened indices between queries in one session
Every open used a fresh Lance session, so the index and metadata caches
died with the request and each query loaded the IVF index again.
--index-cache-mb sizes the shared cache, 512 MiB by default.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* test/s3vectors: PERF_CLUSTERS draws the perf suite's vectors around shared centres
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* test/s3vectors: the index test draws embedding-like vectors
Independent Gaussian vectors have no locality for an IVF index to use, so
recall through one only tracks the fraction of partitions probed; points
around shared centres behave like embeddings. The test must also run
before the one that stops the engine.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* test/s3vectors: the optimize and retrain steps draw embedding-like vectors
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* seaweed-vector: test that compaction drops the deleted key from tied hits
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* seaweed-vector: test that an append-only optimize serves the new row
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* test/s3vectors: record PERF_NOISE in the perf report config
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* seaweed-vector: a filtered query widens to four times the probes, not without bound
With no upper bound Lance prepares every partition for the late search
even when the minimum probes already hold k rows, and a filtered query
through the index cost as much as the exact scan. A query that names its
own probe count gets exactly that count.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* test/s3vectors: report recall through an index without asserting it
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* test/s3vectors: count a hit tying the k-th best score toward recall
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* admin: lance_vector_index runs in the Lance lane
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* weed-worker: check a table's index before counting its rows
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* test/s3vectors: make build and clean cover the worker leg's binaries
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* weed-worker: a retrain rebuilds the index with partitions for the row count
Lance's OptimizeOptions::retrain does not size the partitions: it left a
6,000-row index with one. The retrain goes through the builder with
sqrt(rows) partitions, for the job and for weed-vector-cli index optimize
--retrain alike. The end-to-end test no longer expects a merge to keep the
partition count.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* test/s3vectors: allow float32 rounding at the recall cutoff
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* test/s3vectors: count a repeated hit key once toward recall
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* test/s3vectors: credit cutoff ties only up to the slots the top-k leaves
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* test/s3vectors: assert every filtered hit passes the filter, indexed or not
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* weed-worker: coverage over an index's names, one name extended, a merge at the delta limit
A sharded build leaves one name per shard, each missing the others'
fragments, so coverage is the union of every index on the column and an
optimize gives new rows to the name with the fewest deltas only. At
MAX_DELTAS the policy passes OptimizeOptions::merge(deltas); Lance's
default options may append instead of merging. A retrain keeps an index's
own metric and rebuilds only IVF_FLAT indices.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* weed-worker: the optimize job measures and extends a vector table's indices together
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* seaweed-vector: tests for bounded deltas, disjoint shards and a retrain's metric
Nine append-and-optimize cycles give delta counts 2 to 8 and then 1; two
names on disjoint fragments stay disjoint through an optimize; a cosine
index on a table without metric metadata is still cosine after a retrain.
The end-to-end test expects the appended delta after the first optimize.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* weed-vector: forget the shared session when a table was dropped and created again
The session keys index lists and transactions by dataset version, and a
table created again at the same location restarts its versions. An open
that finds an older version than the newest seen at that URI, or the same
version committed later, replaces the session and opens again.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* weed-vector: test an open after another engine created the table again
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* weed-worker: retrain only a single IVF_FLAT vector index name
The job compared each shard name's partitions with the whole table's
rows, so a sharded index was always due, and a non-IVF_FLAT index made
optimize bail on every run before the new rows were indexed.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* weed-worker: refuse to retrain a vector index sharded over several names
A rebuild per name over the whole table left every shard covering every
fragment, so a query searched each row once per shard.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* seaweed-vector: test which vector indices a retrain is due for
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* weed-vector: replace the session in the same lock that records a recreated table
A second open of the recreated table could see the new version recorded
and the old session still shared, and keep the dropped table's index list.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* weed-worker: skip a vector table whose rows cannot be counted
Detection returned on the first table whose count_rows failed, so one
unreadable table kept every other table from being proposed. It now skips
that table like an open or index failure.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* grafana: panel for vector indexes the Lance worker built
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* seaweed-vector: size default probes from the largest index on the column
A sharded build gives the column one index name per shard. The first
name alone could be a small shard, and its probe count would then be
spent on a larger one.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* seaweed-vector: skip the probe lookup when a query is given no segments
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* weed-vector: open every table in its own Lance session again
One session shared by every open keeps a table's index loaded between
queries, but Lance keys what it caches by location and version, so a
table dropped and created again at the same location could be served
the old table's index list. Telling the two apart from version and
commit time needs a map that only grows, a retry loop and clocks that
agree across engines. A private session per open, as before, has no
such state. Measured at 50k x 384 through an index, eight clients: 164
queries/s with the shared session, 137 without; one client is unchanged.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* weed-vector: reject --nprobes 0
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* seaweed-vector: read the partition count from the opened index
Lance's index statistics took 3.6 ms a query on a 200k-row table with
447 partitions. Opening the index and asking it takes 0.4 ms, and the
scan that follows finds the index already open.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* ci: give the S3 Vectors worker legs the vendored protoc
The worker workspace has no protoc of its own for lance's build
scripts, so the worker legs failed at their first build.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* weed-worker: give a vector index build an hour before the admin times it out
The job set no execution timeout, so the admin's default of 90 seconds
applied, and a build that trains on a large table would be reported as
failed and retried while it was still running.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* weed-worker: lance_vector_index proposes at most the admin's max_results
Every qualifying table went into one message however many there were.
The detector stops at the limit the request carries and sets has_more,
as the Go detectors do.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* weed-worker: a vector index job re-checks the table it was proposed for
A table emptied since detection completes without a build, and one
without a vector column fails instead of reporting it already indexed.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* weed-worker: lance_optimize_indices gets a build's execution timeout
A retrain rebuilds the index over the whole table, which the admin's
default of 90 seconds would cut off on a large one.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* weed-worker: a vector index run window as long as one build
An attempt is capped at the end of the job type's run window plus the
drain grace, and the default window is 30 minutes, so a build could be
cut off before its own hour was up. The window is an hour too, as the
Go iceberg handler sets both.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* weed-worker: a lance_optimize_indices run window as long as one retrain
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* weed-worker: vector index proposals declare a build's runtime
A run started by hand while the job type's schedule is off takes no
timeout from the descriptor, only the admin's 90 seconds. The admin
gives a job at least the runtime its proposal declares, so the build
and the optimize proposals of a vector table declare the hour.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* seaweed-vector: coverage reports no partitions for an index of unknown coverage
Reading the partition count goes through Lance's index statistics,
which fail on an index that records no fragment bitmap, and the whole
coverage report failed with them.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* weed-worker: an index that records no fragments is not read as uncovered
Its rows counted as uncovered, so detection proposed the table and the
execution then failed in Lance's statistics. Coverage fails for such an
index instead, and detection skips the table with a warning.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* seaweed-vector: coverage reads partition counts from the opened index
Lance's index statistics rewrite the manifest of a table whose
fragments carry no row counts, which a coverage report must not do and
cannot do with read-only credentials. The report asks each opened
segment instead, as the probe lookup does, and the statistics-based
reader has no caller left.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
---------
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
* seaweed-vector: build and extend the vector index
weed_lance_worker::vector_index builds an IVF_FLAT index in the metric the
table was created with, sizes partitions from the row count, and extends an
index with a new segment or a retrain; the core crate re-exports it and adds
the probes a query spends. Tests cover coverage after build, append, retrain
and compaction.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* weed-vector-cli: index build and index optimize
Logs go to stderr so stdout stays the JSON a caller parses.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* test/s3vectors: PERF_INDEX builds the index before the timed queries
The cluster fixture hands out its catalog URL, the Makefile builds
weed-vector-cli, and the perf suite builds the index after ingest and
reports the build time and segment count.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* seaweed-vector: probe a tenth of the index's partitions, more under a filter
A query that names no probe count gets one from the index's partition
count. Lance searches at least that many partitions, and under a prefilter
keeps going until it has k rows; without one it stops there.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* weed-vector: --nprobes overrides the probe policy
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* seaweed-vector: test that a build takes the table's metric
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* test/s3vectors: make clean removes weed-vector-cli
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* seaweed-vector: measure recall with vectors the table does not hold
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* weed-vector: keep the read cache's doc comment on with_read_cache
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* weed-worker: lance_vector_index builds the index of an S3 Vectors table past a row threshold
Detection keeps the tables a vector bucket created, with at least min_rows
rows (50,000 by default) and no index on their vector column. Execution
builds the IVF_FLAT index in the table's metric with sqrt(rows) partitions.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* seaweed-vector: optimize only the indices on the vector column
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* seaweed-vector: resolve the query metric with index::distance_metric
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* seaweed-vector: test the L2 metric under the name CreateIndex stores
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* test/s3vectors: build the index through the admin server and the Lance worker
VECTOR_WORKER=1 adds weed admin and weed-worker to the cluster. The new
test sets the job's threshold, runs it through the plugin API, checks
coverage and recall, and that a row written after the build is found.
CI gains the worker leg for both volume servers.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* weed-worker: open a table inside a caller's Lance session
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* weed-vector: keep opened indices between queries in one session
Every open used a fresh Lance session, so the index and metadata caches
died with the request and each query loaded the IVF index again.
--index-cache-mb sizes the shared cache, 512 MiB by default.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* test/s3vectors: PERF_CLUSTERS draws the perf suite's vectors around shared centres
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* test/s3vectors: the index test draws embedding-like vectors
Independent Gaussian vectors have no locality for an IVF index to use, so
recall through one only tracks the fraction of partitions probed; points
around shared centres behave like embeddings. The test must also run
before the one that stops the engine.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* seaweed-vector: test that compaction drops the deleted key from tied hits
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* seaweed-vector: test that an append-only optimize serves the new row
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* test/s3vectors: record PERF_NOISE in the perf report config
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* seaweed-vector: a filtered query widens to four times the probes, not without bound
With no upper bound Lance prepares every partition for the late search
even when the minimum probes already hold k rows, and a filtered query
through the index cost as much as the exact scan. A query that names its
own probe count gets exactly that count.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* test/s3vectors: report recall through an index without asserting it
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* test/s3vectors: count a hit tying the k-th best score toward recall
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* admin: lance_vector_index runs in the Lance lane
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* weed-worker: check a table's index before counting its rows
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* test/s3vectors: make build and clean cover the worker leg's binaries
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* test/s3vectors: allow float32 rounding at the recall cutoff
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* test/s3vectors: count a repeated hit key once toward recall
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* test/s3vectors: credit cutoff ties only up to the slots the top-k leaves
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* test/s3vectors: assert every filtered hit passes the filter, indexed or not
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* weed-vector: forget the shared session when a table was dropped and created again
The session keys index lists and transactions by dataset version, and a
table created again at the same location restarts its versions. An open
that finds an older version than the newest seen at that URI, or the same
version committed later, replaces the session and opens again.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* weed-vector: test an open after another engine created the table again
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* weed-vector: replace the session in the same lock that records a recreated table
A second open of the recreated table could see the new version recorded
and the old session still shared, and keep the dropped table's index list.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* weed-worker: skip a vector table whose rows cannot be counted
Detection returned on the first table whose count_rows failed, so one
unreadable table kept every other table from being proposed. It now skips
that table like an open or index failure.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* grafana: panel for vector indexes the Lance worker built
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* seaweed-vector: size default probes from the largest index on the column
A sharded build gives the column one index name per shard. The first
name alone could be a small shard, and its probe count would then be
spent on a larger one.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* seaweed-vector: skip the probe lookup when a query is given no segments
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* weed-vector: open every table in its own Lance session again
One session shared by every open keeps a table's index loaded between
queries, but Lance keys what it caches by location and version, so a
table dropped and created again at the same location could be served
the old table's index list. Telling the two apart from version and
commit time needs a map that only grows, a retry loop and clocks that
agree across engines. A private session per open, as before, has no
such state. Measured at 50k x 384 through an index, eight clients: 164
queries/s with the shared session, 137 without; one client is unchanged.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* weed-vector: reject --nprobes 0
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* seaweed-vector: read the partition count from the opened index
Lance's index statistics took 3.6 ms a query on a 200k-row table with
447 partitions. Opening the index and asking it takes 0.4 ms, and the
scan that follows finds the index already open.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* ci: give the S3 Vectors worker legs the vendored protoc
The worker workspace has no protoc of its own for lance's build
scripts, so the worker legs failed at their first build.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* weed-worker: give a vector index build an hour before the admin times it out
The job set no execution timeout, so the admin's default of 90 seconds
applied, and a build that trains on a large table would be reported as
failed and retried while it was still running.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* weed-worker: lance_vector_index proposes at most the admin's max_results
Every qualifying table went into one message however many there were.
The detector stops at the limit the request carries and sets has_more,
as the Go detectors do.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* weed-worker: a vector index job re-checks the table it was proposed for
A table emptied since detection completes without a build, and one
without a vector column fails instead of reporting it already indexed.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* weed-worker: a vector index run window as long as one build
An attempt is capped at the end of the job type's run window plus the
drain grace, and the default window is 30 minutes, so a build could be
cut off before its own hour was up. The window is an hour too, as the
Go iceberg handler sets both.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
---------
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
* seaweed-vector: build and extend the vector index
weed_lance_worker::vector_index builds an IVF_FLAT index in the metric the
table was created with, sizes partitions from the row count, and extends an
index with a new segment or a retrain; the core crate re-exports it and adds
the probes a query spends. Tests cover coverage after build, append, retrain
and compaction.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* weed-vector-cli: index build and index optimize
Logs go to stderr so stdout stays the JSON a caller parses.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* test/s3vectors: PERF_INDEX builds the index before the timed queries
The cluster fixture hands out its catalog URL, the Makefile builds
weed-vector-cli, and the perf suite builds the index after ingest and
reports the build time and segment count.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* seaweed-vector: probe a tenth of the index's partitions, more under a filter
A query that names no probe count gets one from the index's partition
count. Lance searches at least that many partitions, and under a prefilter
keeps going until it has k rows; without one it stops there.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* weed-vector: --nprobes overrides the probe policy
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* seaweed-vector: test that a build takes the table's metric
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* test/s3vectors: make clean removes weed-vector-cli
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* seaweed-vector: measure recall with vectors the table does not hold
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* weed-vector: keep the read cache's doc comment on with_read_cache
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* seaweed-vector: optimize only the indices on the vector column
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* seaweed-vector: resolve the query metric with index::distance_metric
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* seaweed-vector: test the L2 metric under the name CreateIndex stores
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* weed-worker: open a table inside a caller's Lance session
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* weed-vector: keep opened indices between queries in one session
Every open used a fresh Lance session, so the index and metadata caches
died with the request and each query loaded the IVF index again.
--index-cache-mb sizes the shared cache, 512 MiB by default.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* test/s3vectors: PERF_CLUSTERS draws the perf suite's vectors around shared centres
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* seaweed-vector: test that compaction drops the deleted key from tied hits
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* seaweed-vector: test that an append-only optimize serves the new row
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* test/s3vectors: record PERF_NOISE in the perf report config
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* seaweed-vector: a filtered query widens to four times the probes, not without bound
With no upper bound Lance prepares every partition for the late search
even when the minimum probes already hold k rows, and a filtered query
through the index cost as much as the exact scan. A query that names its
own probe count gets exactly that count.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* test/s3vectors: report recall through an index without asserting it
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* test/s3vectors: count a hit tying the k-th best score toward recall
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* test/s3vectors: allow float32 rounding at the recall cutoff
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* test/s3vectors: count a repeated hit key once toward recall
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* test/s3vectors: credit cutoff ties only up to the slots the top-k leaves
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* test/s3vectors: assert every filtered hit passes the filter, indexed or not
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* weed-vector: forget the shared session when a table was dropped and created again
The session keys index lists and transactions by dataset version, and a
table created again at the same location restarts its versions. An open
that finds an older version than the newest seen at that URI, or the same
version committed later, replaces the session and opens again.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* weed-vector: test an open after another engine created the table again
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* weed-vector: replace the session in the same lock that records a recreated table
A second open of the recreated table could see the new version recorded
and the old session still shared, and keep the dropped table's index list.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* seaweed-vector: size default probes from the largest index on the column
A sharded build gives the column one index name per shard. The first
name alone could be a small shard, and its probe count would then be
spent on a larger one.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* seaweed-vector: skip the probe lookup when a query is given no segments
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* weed-vector: open every table in its own Lance session again
One session shared by every open keeps a table's index loaded between
queries, but Lance keys what it caches by location and version, so a
table dropped and created again at the same location could be served
the old table's index list. Telling the two apart from version and
commit time needs a map that only grows, a retry loop and clocks that
agree across engines. A private session per open, as before, has no
such state. Measured at 50k x 384 through an index, eight clients: 164
queries/s with the shared session, 137 without; one client is unchanged.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* weed-vector: reject --nprobes 0
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* seaweed-vector: read the partition count from the opened index
Lance's index statistics took 3.6 ms a query on a 200k-row table with
447 partitions. Opening the index and asking it takes 0.4 ms, and the
scan that follows finds the index already open.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
---------
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
* seaweed-vector: build and extend the vector index
weed_lance_worker::vector_index builds an IVF_FLAT index in the metric the
table was created with, sizes partitions from the row count, and extends an
index with a new segment or a retrain; the core crate re-exports it and adds
the probes a query spends. Tests cover coverage after build, append, retrain
and compaction.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* weed-vector-cli: index build and index optimize
Logs go to stderr so stdout stays the JSON a caller parses.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* test/s3vectors: PERF_INDEX builds the index before the timed queries
The cluster fixture hands out its catalog URL, the Makefile builds
weed-vector-cli, and the perf suite builds the index after ingest and
reports the build time and segment count.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* seaweed-vector: test that a build takes the table's metric
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* test/s3vectors: make clean removes weed-vector-cli
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* seaweed-vector: optimize only the indices on the vector column
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* seaweed-vector: resolve the query metric with index::distance_metric
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* seaweed-vector: test the L2 metric under the name CreateIndex stores
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* test/s3vectors: PERF_CLUSTERS draws the perf suite's vectors around shared centres
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* seaweed-vector: test that compaction drops the deleted key from tied hits
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* seaweed-vector: test that an append-only optimize serves the new row
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* test/s3vectors: record PERF_NOISE in the perf report config
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* test/s3vectors: report recall through an index without asserting it
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* test/s3vectors: count a hit tying the k-th best score toward recall
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* test/s3vectors: allow float32 rounding at the recall cutoff
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* test/s3vectors: count a repeated hit key once toward recall
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* test/s3vectors: credit cutoff ties only up to the slots the top-k leaves
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* test/s3vectors: assert every filtered hit passes the filter, indexed or not
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
Both were generated at the repo root, so their embedded source paths read
weed/admin/view/... where every other admin template, generated by
weed/admin's Makefile, reads view/.... No markup changes.
Claude-Session: https://claude.ai/code/session_011UpvafcfUuC2mLxNAXLQJi
* admin/plugin: fail a dispatch its restarted worker no longer lists
A job dispatched to a worker that then restarts dies with the old
process. The worker re-registers under the same id on a new stream and
its heartbeats stop listing the job, but nothing ended the pending
execution: the dispatcher kept waiting and held the executor slot until
the attempt's execution timeout.
Each pending execution now records the stream it went out on. An
itemized heartbeat from a newer stream of that worker that leaves the
job out twice in a row fails it through the expire path, so the
dispatch returns, the slot is released and the retry machinery takes
over. Jobs on the heartbeat's own stream, jobs the new stream still
lists, and heartbeats that count slots without itemizing them are left
alone.
Claude-Session: https://claude.ai/code/session_011UpvafcfUuC2mLxNAXLQJi
* admin/plugin: let a partly itemized listing clear a dispatch's misses
A heartbeat that counted more slots than it itemized returned before the
listed jobs had their misses reset, so an omission, a partial heartbeat
listing the job, and another omission failed a job the worker was still
running.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* admin/plugin: look up the worker's stream before tracking the attempt
A dispatch to a worker with no stream returned after the job was already
recorded as assigned, and that record kept new proposals with the same
dedupe key out until the stale-job sweep expired it.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* admin/plugin: fail the attempt whose request could not be sent
The dispatch now sends on the stream it looked up before tracking the
attempt, so a worker that reconnects in between makes the send fail.
That returned with the job still tracked as assigned, and with the
worker back nothing closed the record for the 24h stale timeout while
its dedupe key kept new proposals out. The attempt is now completed as
failed, after it is unregistered so the completion does not wait on a
channel nobody reads.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* admin/plugin: leave a send failure to the expiry that already claimed it
The send-error path and an orphan expiry could both complete the same
attempt and record two failed runs for it. The send-error path now
completes the attempt only if no expiry has marked it failed, decided
under the same lock that unregisters it.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* admin/plugin: record a send failure whose entry Shutdown already removed
Shutdown closes shutdownCh before it unregisters pending dispatches, so a
blocked send returns while its map entry is gone. Read the expiry claim
from the dispatch's own state instead of the map lookup, so the attempt
is still failed and does not stay assigned.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* admin/plugin: match a padded job id against its trimmed listing
Heartbeat work ids are trimmed before the orphan check, so compare the
dispatch's job id trimmed too; otherwise a job id with surrounding
spaces never matches its own listing and the running job is failed.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* admin/plugin: skip orphan misses once a completion was handed over
The buffer length only shows a completion the dispatcher has not taken
yet; after it takes one, the attempt stays registered until the dispatch
returns. Mark the hand-off under the pending lock and check that flag
instead, so heartbeats in that gap cannot fail a finished attempt.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* seaweed-volume: report EC file and delete counts in heartbeat deltas
The full EC heartbeat already carries file_count and delete_count, but the
incremental new/deleted shard messages built in
collect_ec_shard_delta_messages left both at zero. A volume the master first
learns of through a delta then reads 0 files until the next full EC
heartbeat. Fill both from file_and_delete_count, the same source the full
heartbeat uses, computed once per volume rather than once per shard.
Claude-Session: https://claude.ai/code/session_011UpvafcfUuC2mLxNAXLQJi
* volume: report EC file and delete counts in mount notifications
Go parity with the Rust delta change: the NewEcShardsChan messages sent on
MountEcShards and on startup shard load now carry FileCount and DeleteCount
from EcVolume.FileAndDeleteCount, the same values the full EC heartbeat
reports, instead of leaving them at zero.
Claude-Session: https://claude.ai/code/session_011UpvafcfUuC2mLxNAXLQJi
* seaweed-volume: leave count-only EC changes out of the shard delta
The delta messages now carry file_count and delete_count, so the whole-message
comparison re-sent every shard of a volume after any delete, and the master
broadcast each one as a new EC vid. Compare the shard fields only; the full EC
heartbeat already carries the counts.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* topology: mount messages now carry EC counts, only older servers omit them
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* storage: guard the captured EC mount messages in the disk-type tests
The drain goroutine appended to the slice that waitForShardMsg polled with no
lock, which go test -race reports.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* storage: test EC counts on the startup notification and repeated .ecj ids
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* seaweed-volume: move file_count too in the count-only delta test
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* shell: accept ReadAcp and WriteAcp in per-bucket and service account actions
s3.bucket.access, s3.anonymous.set and s3.serviceaccount.create rejected
ReadAcp and WriteAcp, so the ACL actions the S3 gateway enforces could not
be granted through them. Share one canonical action table across the
three commands and list the ACL actions in their help.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* shell: list ReadAcp and WriteAcp in s3.configure and s3.circuitBreaker help
The -actions usage of both commands stopped at Admin, so the ACL actions
the gateway enforces could only be found in the source. s3.circuitBreaker
now prints the actions it actually accepts.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* admin: offer ReadAcp and WriteAcp in the object store user permissions
The create and edit user forms only offered Read, Write, List, Tagging and
Admin among the bucket permissions, so an identity could not be given the
ACL actions from the UI even though uploads with x-amz-acl need WriteAcp.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3api: log the WriteAcp denial of an upload that sets an ACL
An upload or copy that carries x-amz-acl or an x-amz-grant-* header also
needs WriteAcp. When that check denies, the client only sees a bare
AccessDenied and nothing reached the log at the default verbosity, so
the header was never suspected. Log the identity and the missing action.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3api: quote the key in the ACL upload denial log
An object key can carry a decoded newline, which let a caller add forged
lines to the default log. Quote the bucket and key, and word the message
so it also fits a policy deny, not only a missing grant.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* shell: say ACL uploads need both Write and WriteAcp
The s3.configure example granted WriteAcp alone, which still fails the
upload's Write check.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* shell: parse S3 action lists with one shared helper
s3.bucket.access, s3.anonymous.set and s3.serviceaccount.create each had
their own validation loop, so a test of the table did not cover the
parsing the commands run. They now call canonicalActionList, which skips
empty entries and repeats. s3.anonymous.set validates before contacting
IAM, and s3.bucket.access still rejects a list with no actions.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* admin: keep bucket-scoped actions the user form cannot show
When a user has both global and bucket-scoped actions, the edit form
shows only the global ones, and saving replaced the identity's actions
with what the form rebuilt, deleting the scoped grants. Carry those
actions through the save.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* admin: list hidden bucket grants in the user edit form
Restoring the bucket-scoped grants the form cannot show kept them active
even when an admin narrowed the user's permissions, with no way to see
or drop them. Show them as checked boxes under Other Bucket Permissions;
unchecking one removes it on save.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* shell: reject an s3.anonymous.set -access list with no actions
A blank list such as ",," parsed to nothing and cleared the bucket's
anonymous grants. Use "none" for that, as s3.bucket.access does.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3api: skip the ACL upload denial log for anonymous requests
Anonymous uploads with ACL headers can be sent by anyone, so logging
each denial at the default level lets unauthenticated clients flood the
log. Keep the line for named identities.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* admin: count kept bucket grants when checking an edited user for permissions
The empty check ran before the checked Other Bucket Permissions were
merged, so a user left with only those grants could not be saved.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* admin: list bucket grants of Admin users in the user edit form
parseBucketPermissions returned before reading scoped actions when the
user had Admin, so removing Admin and saving dropped grants such as
WriteAcp:bucket.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* admin: keep unpaired bucket grants as separate checkboxes in the user edit form
The form saves every selected permission on every selected bucket, so
grants such as ReadAcp:a with WriteAcp:b came back as all four pairs.
Show the grants through the picker only when they cover that full
product; otherwise list each grant on its own. Opening the form also
resets the bucket scope left over from the previous user.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* admin: keep grants the edit pickers have no option for
A permission x bucket product was shown through the pickers even when a
bucket was not in the bucket list (deleted, prefix or wildcard grants)
or a permission had no picker option (DeleteBucket, PutBucketQuota).
Saving then dropped those grants. Such grants now go to the checkbox
list, global ones included, and are submitted unchanged.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* shell: name the s3.bucket.access anonymous example by what it grants
Read,Write,List leaves out ReadAcp and WriteAcp, so it is not full
access.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* admin: keep unlisted global grants of Admin users in the user edit form
parseBucketPermissions returned for Admin before splitting global grants
into picker options and the rest, and only raw option values were
re-selected. Saving an Admin user dropped s3tables:* and grants with no
picker option such as DeleteBucket. Admin users now use the same split.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* admin: keep the /* suffix of bucket grants in the user edit form
parseBucketPermissions dropped a trailing /* before matching bucket
options, so saving rewrote ReadAcp:bucket/* as ReadAcp:bucket. The bare
form also matches bucket-level requests such as GET ?acl, so the save
widened the grant. A /* bucket is not a picker option, so the grant now
stays in the checkbox list as written.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* shell: test that s3.bucket.access saves ReadAcp and WriteAcp grants
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
A Content-Type form field such as "\n" was copied verbatim into the
multipart part header sent to the volume server, which broke the
framing and surfaced as a Content-MD5 mismatch and a 500. Form fields
promoted to request headers must be valid header values, as on a PUT;
otherwise answer 400 MalformedPOSTRequest.
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* fix(s3): accept fresh admin tokens when context carries stale authorization (#12390)
Signed-off-by: Mohd Quamar Tyagi <104281681+Tyagiquamar@users.noreply.github.com>
* security: share the admin Bearer check between S3 and admin-plane peers
Move the authorization scan into security.CheckGrpcAdminToken, built on
IsFilerAdminToken, and call it from CheckGrpcAdminPeer and the S3 IAM cache
check. Both now accept a valid admin token in any authorization value
instead of S3 re-implementing the decode and claim checks.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* filer: accept a fresh admin token after a stale one on IAM gRPC
The S3 IAM cache check is meant to mirror the filer's, so route the filer's
checkAdminAuth through security.CheckGrpcAdminToken too. IamGrpcStore appends
its token to the caller's context, so a forwarded older token would otherwise
still shadow it here.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* security: parse at most four authorization values per admin call
Scanning every value let one request repeat a junk Bearer token (cheap over
HPACK) tens of thousands of times and burn about half a second of JWT parsing
on an unauthenticated port. Real callers send their own token plus at most one
forwarded from their context.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* credential: match the other WithS3InternalAdminAuth replace fix
The plugin worker fix makes the same set-instead-of-append change to
WithS3InternalAdminAuth with different code and a different test at the same
spot, so whichever lands second would conflict. Use the identical code and
test so the two merge cleanly in either order.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* security: test an expired admin token ahead of a fresh one
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* security: test that a same-key session token is not an admin token
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* filer: test IAM gRPC with only invalid authorization values
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Signed-off-by: Mohd Quamar Tyagi <104281681+Tyagiquamar@users.noreply.github.com>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* storage: search corrupt shard sets when an EC needle fails to rebuild
A needle whose reassembled bytes fail to parse is rebuilt with its own
shards excluded. When other shards are corrupt over the same range, that
decode reads them as inputs, and the retry dropped only one input at a
time, so two or more extra corrupt shards left the needle unreadable even
within parity.
After the single pass fails, try every set of up to ParityShards shards
that holds a suspect as the corrupt set, smallest first. Suspect ranges
inside the set are rebuilt from the shards outside it and the rest keep
their bytes, so a clean suspect can still feed the decode. Sets whose
decodes read the same inputs are tried once, and the search is bounded
by set count and by rebuilt bytes past the single-shard sets.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* seaweed-volume: search corrupt shard sets when an EC needle fails to rebuild
Mirror the Go read path: after the single pass over the suspects fails,
gather each interval from every other shard once and try each set of up
to parity_shards shards holding a suspect as the corrupt set, smallest
first, with the same dedup and bounds. The search runs on the blocking
pool, holding the recovery permit, since it can take many decodes.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* ec: charge only multi-shard sets against the rebuild byte budget
The budget check already let single-shard sets through, but each one
was still charged, so a needle spanning many shards could spend the
budget before the first multi-shard set.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* ec: bound the corrupt-set gather to the recovery budget
The search gathered every other shard per interval, so a large needle
spanning several shards held up to Total-1 copies of itself under a
permit capped at the budget. Gather only as many as the budget covers,
but never fewer than the suspects-excluded gather the fallback always
held, so single-suspect recovery keeps every other shard.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* ec: gather at least one decode's worth for the corrupt-set search
A needle over more shards than parity skips the single pass, and its
suspects-excluded width can fall below DataShards, so the bounded
gather left even one corrupt shard unrecoverable. Floor the width at
DataShards, which the per-suspect rebuild this search replaced read.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* ec: rebuild one suspect at a time when the search gather is narrow
Flooring the search gather at DataShards made a large needle over more
shards than parity hold ten copies of itself, where the per-suspect
rebuild it replaced held one interval's fan-out. Keep that rebuild
after the search, and skip the search when the budget leaves fewer
inputs than one decode. Parity slack still widens the gather to every
shard outside the suspects, as the dropped-input retry did.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* ec: keep the dropped-input retry and run the set search within budget
A narrowed search gather skipped only each interval's own shard, so it
could read corrupt suspects in place of healthy shards the old retry
kept, and its byte bound could stop before that retry's hypotheses.
Restore the dropped-input retry as it was, move the search into its
own function after it, and run the search only when every interval's
full gather fits the recovery budget. Past the budget a read takes the
same paths, with the same footprint, as before the search.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* grafana: chart EC intervals served by reconstruction
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* grafana: chart volume server storage IO errors and quarantine
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* grafana: chart frozen volume growth and duplicate volume ids
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* grafana: chart filer metadata log gaps and replay failures
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* grafana: add failed bytes to filer sync throughput
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* grafana: chart remote mount cache reads and evictions
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* grafana: chart plugin worker detection duration
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* grafana: show duplicate volume ids whose first detection was the only one
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* grafana: keep cluster apart in the master growth and duplicate id panels
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* volume: count EC reads that fall back to reconstruction
When a needle read can't read an EC shard interval directly (the shard isn't
mounted here and no holder answers), both volume servers rebuild it from the
other shards and the read still succeeds, with nothing to show for it.
Count those fallbacks in SeaweedFS_volumeServer_ec_degraded_read_total{result}
(success, failure), one per interval, at the one place each server falls back:
Go readOneEcShardInterval, Rust fetch_one_interval. Corruption repair and scrub
aren't counted. A line binding each one to the needle and the volume's encode
generation is logged at V(3) in Go and debug in Rust.
* volume: name the EC counter for what it counts
Reed-Solomon reconstruction can't tell corrupt inputs from good ones, and the
needle is validated later, so "success" only ever meant the reconstruction
produced data. Rename the counter to
SeaweedFS_volumeServer_ec_reconstructed_interval_total and the log line to
"ec reconstructed interval", say so in the help text, and pin the behaviour
with a test: an interval rebuilt from corrupt parity counts success while the
read itself fails.
* storage: restore stderr when the log-line test stops early
A fixture failure in readEcNeedle ends the test through t.Fatalf, which
skipped restoring os.Stderr and closing the pipe, so later tests wrote
into a pipe nobody read.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* s3tables: validate policy conditions and fail safe on unusable ones
* s3vectors: reject a condition operator with no keys
The shared evaluator now treats a keyless operator as unusable, so a
stored keyless Allow would never grant while the write succeeded.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3vectors: reject an Allow whose Condition is null
A null Condition decodes as no Condition, so the Allow granted
unconditionally. A null Condition on a Deny only widens the veto and
stays accepted, as in the S3 Tables validator.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3tables: treat a NaN numeric condition value as unreadable
strconv.ParseFloat accepts "NaN", but no comparison with it holds, so a
Deny on NumericLessThan never applied and an Allow on NumericNotEquals
matched every request.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3tables: reject condition keys the evaluator never supplies
A condition on a key such as aws:SourceIp reads no request value, so a
Deny on it never applied and an Allow never granted. Accept only the
keys getConditionContextValues resolves, as the vector validator does.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3vectors: require Statement and Effect spelled as written
The evaluator decodes field names regardless of case, so an "effect" or
"statement" key slipped past the null Condition check and the other
raw-key checks while the stored Allow still granted.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3tables: treat an infinite numeric condition value as unreadable
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3tables: a stored Allow with a null Condition never grants
The typed decode reads "Condition": null as no Condition, so an Allow
stored before validation refused it still granted unconditionally, for
table and vector policies alike. Record the literal null while decoding
a Statement and skip such an Allow during evaluation.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3tables: validate a null Condition from the decoded Statement
validatePolicyDocument re-parsed the document to find a literal null
Condition; the Statement decode now records it, so the validator reads
the same flag the evaluator does and nullConditionStatements goes away.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3tables: keep Condition objects merged across key spellings
Decoding Condition through a json.RawMessage kept only the last of
"Condition" and "condition", so an empty object under one spelling
erased the other's restriction. Decode the statement as before and
probe for the literal null separately.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3tables: refuse a Condition split across key spellings
Condition objects under differently cased keys decode into one map, and an
operator they share keeps only the last spelling's keys, so a restriction
under the other spelling was dropped. Validation now refuses a statement
whose Condition is under more than one key, and one stored before that
never grants on an Allow and vetoes on a Deny.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3tables: refuse a Condition key repeated under the same spelling
PolicyDocument re-marshaled each statement through interface{}, and
Statement.UnmarshalJSON counted keys from a map, so both kept only the
last of a repeated Condition key. "Condition":null,"Condition":{} or two
Condition objects sharing an operator passed validation with a restriction
dropped. Statements now decode from their original bytes and the keys are
counted while walking the object.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3vectors: refuse a repeated Condition key in a vector bucket policy
The validator read Condition from a map, which keeps only the last of a
repeated key, so it accepted a statement the evaluator never grants on.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3tables: keep the re-marshaled statement decode, count Condition keys apart
Decoding statements from their original bytes let document order pick
between spellings of every field, where the re-marshal sorted the keys
first, so a stored "effect":"Deny","Effect":"Allow" became an Allow.
Statements decode as before again; only the Condition keys are counted
from the original bytes.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3tables: test that validation refuses a split Condition on a Deny
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3vectors: test that validation refuses a repeated Condition on a Deny
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* plugin worker: keep the stream's admin token out of job contexts
The bearer token minted when the worker opens its control stream was
attached to connCtx, which every detection and execution context derives
from. Job RPCs therefore forwarded that connect-time token, and when a
job added its own fresh token (s3_lifecycle's LifecycleDelete) the S3
server validated the stale one first and rejected the call once it
expired.
Attach the token only to the WorkerStream call, as the Rust worker does.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* credential: replace an inherited admin token instead of appending one
WithS3InternalAdminAuth appended its fresh Bearer token, so a context
that already carried an authorization value sent both, and the S3
gateway's checkAdminAuth validates only the first. Set the header on a
copy of the outgoing metadata so the fresh token is the only one sent.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* plugin worker: read the captured job metadata after the stream ends
Both result channels could be ready at once, letting select fail a
successful run. The handler reports before JobCompleted reaches admin,
so wait for the stream to finish and then read the captured metadata.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* plugin worker: test that dropping the connection cancels running jobs
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* s3: keep service account keys when a pushed identity replaces its parent
Service account changes push the parent user, whose proto carries only
the user's own keys. The merge replaced the cached parent with that
proto and dropped every service account key until a restart. Read the
parent's service accounts from the credential store so the push
rebuilds them, deleted and disabled ones included.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3: test that a parent identity push keeps its service account keys
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3: test a sibling service account survives a deletion push
Also check the restored secret and status, not just the key.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
seaweed-worker-core now depends on toml, but the seaweed-vector
workspace lockfile was not refreshed, so cargo build --locked fails
before the S3 Vectors integration tests start.
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* filer: clear a dead test replica's reachability record before closing it
The read path tries hosts that recently failed last. A dead test server
can land on a port an earlier test left marked as failed, so the live
replica answered first, nothing failed and the location refresh under
test never ran. Have the dead replica answer once through util_http so
its record is cleared before it closes.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* filer: close the dead test replica when reaching it fails
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* filer: clear the live test replica's reachability record too
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* master: tell a warming-up Assign client when warm-up ends
During warm-up the master refuses Assign with Unavailable and now attaches
the remaining warm-up as a gRPC RetryInfo detail. RetryWithBackoff may
shorten a backoff step to that hint plus 50 ms, never lengthen it, and only
for a valid, positive hint, so a writer retries right after warm-up ends
instead of up to one backoff step later. The warm-up itself is unchanged.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* master: jitter the retry-hint slack and hint LookupVolume too
Per review: the slack added to a server hint is now random in
[50ms, 300ms], so clients refused at the same moment do not all retry
in one instant. The warm-up refusal of LookupVolume carries the same
RetryInfo hint as Assign; the masterclient's lookup retry already goes
through RetryWithBackoff, so it lands at warm-up end too.
Tests: the slow guard tests are replaced by one sleep-free table test
of the step logic (zero, negative, huge, malformed, valid and wrapped
hints, a hint longer than the step, none). One timing test of the
shortened step stays, plus a jitter-bounds test and a LookupVolume hint
test. The no-hint control test, which was timing-sensitive, is dropped.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* master: name the shared warm-up test and file the lookup test by topic
Rename the no-remaining test to TestWarmingUpErrorWithoutRemainingHasNoHint,
since the helper now serves Assign and LookupVolume, and move the
LookupVolume hint test next to the other LookupVolume warm-up tests.
Tests only.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* util: read the retry hint through ServerStatus
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* master: bound the warm-up hint in tests by the remaining warm-up around the RPC
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* test: leave more scheduler margin in the retry-hint timing tests
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* util, master: trim retry-hint comments
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* test: widen the retry-hint timing bounds to just under the unhinted wait
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* test: require a refused warm-up call before the hinted assign retry
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* test: check the hinted retry wait through a test timer, not the wall clock
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* test: move the assign warm-up end to 2.6s for more hinted-retry headroom
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* test: require the hinted retry to wait out its timer
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
* admin: credit an unitemized slot to a pending job only after a later heartbeat
A pending reservation the worker does not list took one of the slots
the worker counts without itemizing, even when that heartbeat came
before the reservation and could not have counted the job. The slot
belonged to unknown work, so the admin undercounted it and could admit
one job too many, past a job type's per-worker limit, until the next
heartbeat.
A reservation now records when it was made, and claims an unitemized
slot only from a heartbeat received at or after that time.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* admin: time unitemized-slot credit by the heartbeat, not the last hello
A reconnect's hello moves LastSeenAt but keeps the old heartbeat, so a
pending reservation could claim a slot counted before it existed, and a
completed job's echo could end without a new heartbeat. Record when the
heartbeat itself arrived and compare reservations against that.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* admin: a completed job claims no slot from a heartbeat before its reservation
A job that completes before any heartbeat counts it has no echo, so the
slot the last heartbeat counted is other work. Crediting it to the job's
type let another job type pass its per-worker limit until the next
heartbeat.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* admin: count executor slots per job type, deduped against the heartbeat
per_worker_execution_concurrency is a per-job-type setting, but the
scheduler compared it against the worker's execution_slots_used, which
counts every job type. One s3_lifecycle job a worker reported running
after a reconnect made admin_script (per-worker 1) undispatchable on a
worker with three free slots. A reservation and the heartbeat entry for
the same job were also charged as two slots, so a per-worker limit of N
let N-1 jobs run.
Reservations now carry the job id and job type. The per-type limit is
checked against that type's work alone, the worker's slot total against
work of every type, and a heartbeat entry covered by a reservation is
counted once. Per-lane reservation pools go away: the per-type count is
what kept lanes from starving each other.
* admin: stop a lane waiting on executor capacity it cannot free
A queued job no executor could take was retried every backoff until
job_type_max_runtime ran out, up to 30 minutes, while the default lane
sat in executing and every other job type in it idled.
The dispatch loop now asks what holds the capacity. Slots held by the
lane's own jobs release when they complete, so it keeps waiting. Slots
held by other lanes or by work this admin never dispatched clear on a
schedule the lane has no view of, so the queued jobs are canceled at
once with a skipped_no_executor activity naming the worker, its slot
count and the job types holding it, and the next detection proposes
them again. A missing worker is waited for only the orphan grace.
* admin: cancel a job on its worker when the admin expires it
Expiring a job, by the stale sweep or by the operator's expire endpoint,
only failed the tracked record. The worker never heard about it and kept
running the job with its slot taken, so a job stuck in "starting" held a
slot for over a day and, with the old slot accounting, blocked a whole
lane.
Expiring a job now also sends the worker a cancel for it, so the slot
comes back on the next heartbeat.
* admin: count a completed job's heartbeat echo as its lane's own slot
The worker sends JobCompleted before the heartbeat that frees the slot, so
right after a dispatch goroutine releases its reservation the job still
shows in the worker's running work. That entry was counted as foreign, and
a lane with per-worker 1 gave up and canceled the rest of its queue after
the first job. A released reservation now stays for completedJobEchoGrace
and claims that heartbeat entry as the lane's own load.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* admin: let the admin's own jobs claim slots a worker does not itemize
The Rust worker reports only a slot count, every 10s, and the Go worker
counts a slot one heartbeat before it lists the job. Those slots were
charged on top of the reservations behind them, and once a job completed
its stale count read as foreign work, so the lane gave up its queue.
Unitemized slots now go first to this admin's unlisted reservations and
recent completions; the rest are of unknown type and count against every
type's per-worker limit, not only the slot total.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* admin: give up pinned jobs whose busy home a free worker cannot relieve
A job homed on a replica-host worker stays on that home while its queue is
shorter than LocalQueuePin, so a free remote worker cannot take it. The
outlook treated that free worker as a reason to wait, and a home held by
foreign work parked the lane for its whole window. A free worker now only
holds the wait for the orphan grace, after which the held homes decide.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* admin: let a redispatched job's reservation replace its released one
A job the worker turns away for capacity is released, kept as an echo, and
queued again under the same id. Once the new attempt ran, both the echo and
the new reservation matched its heartbeat entry and it counted as two
slots. Reserving a job now drops what an earlier reservation of it left.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* admin: credit a count-only slot to a completed job only until the next heartbeat
A released reservation claimed an unitemized slot for its full 30s grace,
so on a count-only heartbeat other work of another type was charged to the
completed job's type. A heartbeat that arrives after the release already
reflects the freed slot; from then on the slot is of unknown type.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* admin: name unitemized slots in the no-capacity reason
holders() listed only the job types the worker itemized, so a worker with
named and count-only work read as "2/2 slots used by <type>".
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* admin: keep the blocked clock running until a reservation succeeds
A popped job whose reservation then failed reset the clock before the
failure pushed it back, so pop-then-fail cycles never aged the block.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* admin: resend an expiry cancel the worker did not receive
Expiring a job sends its worker one cancel. When the worker is
disconnected, or its outgoing buffer stays full past the send timeout,
that cancel is dropped. The record is already failed, so later sweeps
and expire calls skip it, and a worker that kept the job running (a
detached job across a reconnect, or a jammed stream) holds the slot
until the job ends on its own.
The admin now holds an undelivered expiry cancel and sends it again on
the worker's next heartbeat that still lists the job. It is forgotten
when that heartbeat no longer lists the job, and skipped when the job is
active again, since a new attempt reuses the job id.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* admin: cancel an expired attempt by its request id
The expiry cancel named the job id, and a retry reuses it. The worker
keys its cancel funcs by both ids, so a cancel arriving after a retry
started on the same worker stopped the retry. The held resend also gave
up on any job that was active again, and the expired attempt's own late
progress marks the job running, so a worker that buffered progress
while disconnected kept the expired job.
The cancel now names the attempt's request id, as the other execution
cancels do, and the held resend no longer depends on the tracked state.
Only a cancel that can name nothing but the job id still skips a job
that is active again.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* admin: ignore late reports from an attempt the admin already ended
Expiring a job now cancels it on its worker, and the worker answers
with its own JobCompleted for that attempt. That overwrote the expiry's
failure with the cancel error and logged a second run record under the
same run id. Late progress from the expired attempt likewise reopened
the record as running.
A progress update or completion whose request id names an attempt the
record already closed is now dropped after the dispatcher's waiter is
told. A retry runs under a new request id, so its reports still apply.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* remote.uncache: skip a file overwritten since the walk read it
The uncache cleared the chunks of the entry its directory walk returned and
wrote it back unconditionally. A file overwritten after the walk read it
was reset to the remote pointer, and its new chunks were dropped, so an
acknowledged write was lost. Write back only while the filer still holds
the entry as read, and report the file as skipped otherwise.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* remote.meta.sync: keep files written locally after the directory listing
meta.sync lists each local directory once and works from that listing.
A changed remote file was written back over the listed entry
unconditionally, and a name missing from the listing was created without
O_EXCL, so a file written locally in between was replaced by the remote
pointer either way. Update only while the filer still holds the listed
entry, create exclusively, and report such files as skipped. The create
now also surfaces errors the filer returns in its response.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* remote.cache: keep files written locally after the lookup
The metadata step looks each file up and then either rewrites it from the
remote unconditionally or creates it without O_EXCL, so a local write
landing between the lookup and the write was replaced by the remote
pointer. Reuse doSaveRemoteEntry, which updates only while the filer still
holds the looked-up entry, create exclusively, and report such files as
skipped.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* remote.meta.sync: do not descend into a name a local file took
When the exclusive create of a remote directory finds the name taken, the
sync still descended into it, and creating the remote children under a
local file would turn that file into a directory. Descend only when the
name now holds a directory.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* remote.meta.sync: delete only the file version it listed
The type-change and -deleteStale paths removed a remote-backed file
unconditionally, so a local write landing after the directory listing was
deleted with it. Condition the delete on the listed mtime, as fs.verify
and volume.fsck do; the filer then keeps a file written since.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* remote.meta.sync: condition directory and mtime-less deletes too
The listed-mtime cutoff was only sent for files, so a remote-backed
directory replaced locally by a file between the emptiness check and the
delete was removed unconditionally, and so was a file listed without an
mtime. Send the cutoff for directories as well, and for an entry listed
without an mtime keep it once a write has set one.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* filer: test that DeleteEntry applies the request's mtime cutoff
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* remote.meta.sync: keep the listed nanoseconds as the cutoff of an epoch mtime
An entry listed at the epoch with a sub-second mtime was sent a 1 ns
cutoff, so the filer kept it unchanged although the old code deleted it.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* remote.meta.sync: report an entry the filer kept instead of deleting it
The filer answers a delete it skipped under the mtime cutoff with
success, so the sync printed (delete) for an entry a local write kept.
deleteLocalEntry now looks the name up after the delete and prints the
outcome, and a replaced file that was kept is not recreated.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* s3api: store and serve a bucket's PublicAccessBlock configuration
PutPublicAccessBlock, GetPublicAccessBlock and DeletePublicAccessBlock
answered NotImplemented. Store the configuration in the bucket entry's
extended attributes next to the ownership controls, carry it in the
cached BucketConfig, and answer Get on an unset bucket with
NoSuchPublicAccessBlockConfiguration like S3 does.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3api: refuse public ACLs when a bucket blocks them
With BlockPublicAcls set, a PutObject, CopyObject, CreateMultipartUpload,
POST Object, PutObjectAcl or PutBucketAcl whose canned ACL or grants name
the AllUsers or AuthenticatedUsers group is refused with AccessDenied.
Private and account grants keep working, so clients that send
x-amz-acl: private on every upload are unaffected.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3api: ignore public ACL grants for anonymous reads under IgnorePublicAcls
A bucket with IgnorePublicAcls no longer lets an AllUsers grant on the
bucket or on an object answer an anonymous read or listing. Bucket
policies and identity permissions are unaffected.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3api: reject PublicAccessBlock bodies with unknown elements
A misspelled element such as BlockPublicAcl was ignored, so the request
succeeded and stored the flag as false. Answer MalformedXML instead, and
store the configuration without the request's namespace.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3api: fail closed when a bucket's public access block cannot be read
A stored PublicAccessBlock that does not parse now blocks and ignores
public ACLs instead of acting as if none were set. PutBucketAcl and POST
Object return the bucket lookup error rather than skipping the
BlockPublicAcls check when the bucket config cannot be loaded.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3api: test BlockPublicAcls on copy, multipart initiate and a PUT-then-GET
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3api: keep stored public access block bytes on unrelated bucket updates
An unreadable stored configuration was replaced with the in-memory
all-true fallback whenever another bucket setting was updated, losing
the original value. Write the key only when the update changes the
public access block.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* volume: a read plan waits for a vacuum commit instead of answering 404
A vacuum commit closes the needle map, swaps the compacted files in and
reloads the map. It holds the data-file write lease for that and takes no
store lock. needle_read_plan, which the HTTP GET path uses, read the needle
map without the data-file read lease, so a GET planned in that window found
no map and answered 404 for a live needle.
needle_read_plan now takes the read lease first, as read_needle_with_option
does, so the plan waits for the commit and then reads the reloaded map.
Go's readNeedle does the same under dataFileAccessLock.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* volume: plan a read with the store guard released
A plan that waits on a vacuum commit's data-file lease under the store
read guard parks store writers (allocate, mount, EC ops) on that commit,
and every store reader queued behind them. Look the volume up under the
guard and plan after releasing it, as Go's readNeedle does, and look the
volume up again if its handle was closed while the plan waited.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* volume: retry a read plan while its handle keeps closing
A plan that waited out an unmount looked the volume up only once more,
so a replacement handle that also closed before its plan took the lease
answered NotFound for a needle a later mount holds. Repeat the lookup
while the planned handle is closed, at most three times, so a closed
handle that stays in the map still ends the loop.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* volume: start the swap-window check once the plan holds the volume
The swap-window test gave the plan 300 ms to answer early, so a reader
thread that started late passed the check without waiting on the
commit. Share the writer test's wait for the plan's volume handle and
start both timed checks after it.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* volume: run the swap-window read-plan test on a redb map too
A commit reloads the index through load_index_redb for the redb kinds,
a path the in-memory run never took during the swap window.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* volume: say a needle read plan is resolved under the data-file lease
The plan no longer resolves under the store guard, and resolving it
takes the data-file read lease.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* volume: read the needle through the plan in the read-plan tests
The tests checked only that the plan resolved and then read the needle
through Store::read_volume_needle, so a plan with a wrong offset or size
after the commit still passed. Read through the plan, as GET does, and
compare the data.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* volume: test the read plan's lookup after an unmount closed the handle
No test reached the closed-handle retry, so dropping it left every test
green. Take the lookup as a closure so a test can hand the plan the
handle an unmount closed and then the remounted volume, and check that
the plan reads the needle from the remount.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
* seaweed-common: share the security.toml search path with the volume server
* weed-worker: read jwt.filer_signing from security.toml and WEED_JWT_* like the Go worker
* helm: give the worker-lance container security.toml and the extra environment
* weed-worker: treat an empty signing key in the environment as unset
Viper ignores an empty environment value, so the Go worker falls back to
the security.toml key when WEED_JWT_FILER_SIGNING_KEY is blank. The Rust
worker took the blank value and connected with no token. An empty
--filer-signing-key / WEED_FILER_SIGNING_KEY also falls through to the
security.toml lookup now.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* weed-worker: read a quoted expires_after_seconds like viper
The Go worker reads jwt.filer_signing.expires_after_seconds with GetInt,
which also accepts a quoted number. The Rust worker dropped a quoted value
and minted tokens with no expiry.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* weed-worker: warn when security.toml cannot be read or parsed
Viper logs a warning when an optional config file is broken. The Rust
worker skipped it silently, so a typo looked like an install with no
signing key. Log the path and the parse message, without the quoted line,
which can hold a key.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* helm: keep POD_NAME and LANCE_NAMESPACE_URL out of worker-lance's extra environment
A duplicate name renders after the chart's entry and wins, silently
changing the Lance worker's --id or --namespace.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* weed-worker: list /usr/local/etc/seaweedfs in the security.toml search path
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* weed-worker: an explicit --filer-signing-key-expires-after-seconds wins
The flag was silently ignored whenever the key was discovered, from
WEED_JWT_FILER_SIGNING_KEY as well as from security.toml. Use it when
given, else the expiry configured beside the discovered key, else zero.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* weed-worker: test where filer_signing looks for security.toml
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* weed-worker: read a float expires_after_seconds as viper's GetInt does
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* weed-worker: cap the minted exp so a huge expiry cannot overflow
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
The fixture waited for the Lance catalog port and the engine, but weed s3
opens its HTTP port last, about half a second after the catalog here. The
open cluster's engine now registers within that window, so
test_engine_without_credentials met a connection refused on the go and rust
single legs. Wait for the S3 port like the other listeners.
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
The detection waiter selects on the request's completion channel and on
the stream's done channel. A worker that finishes a scan and then drops
its stream (a restart right after a scan is the usual shape) buffers the
completion into the request before the receive loop sees the EOF, and
the stream's cleanup then closes done. A waiter that reaches its select
after both are ready can take the disconnect branch, and a completed
detection, proposals included, is failed as "detection worker
disconnected" and retried or recorded as an error.
When the stream closes or the liveness tick finds the worker gone, the
waiter now takes a completion that is already queued and returns it as
the success it is; the detector lease still goes, since the worker that
answered is unreachable. A ctx that ended keeps its meaning: the caller
stopped wanting the answer. The wait loop moves into
awaitDetectionOutcome so a test can hand it both channels already
ready; before this change that test fails on about half its runs. A
completion that lands after the waiter gave up stays unmatched, since
the request's pending entry is gone and the next request has a new id.
* topology: refresh EC file and delete counts from heartbeats
UpdateEcShards only re-stored a heartbeated EC record when shard
membership changed, so FileCount and DeleteCount stayed at whatever the
first heartbeat after registration carried, while the volume server
reports fresh tombstone counts on every full EC heartbeat. Per-collection
object counts and EC vacuum candidate detection read the stored values.
Compare the counts per (volume, disk) against the record already in
hand and copy them onto the stored record in place, so the refresh
raises no new/deleted shard events and rebuilds no maps. A message
reporting 0/0 did not count (older volume servers, and the incremental
mount message omits the fields), so it keeps the stored values, also
across a membership-triggered rewrite. The incremental path's existing
branch picks counts up the same way.
* topology: swap EC shard records instead of mutating them in place
GetEcShards hands out the stored *EcVolumeInfo pointers and releases
ecShardsLock; CollectionStatistics, CollectionEcVolumeStats, ToDiskInfo
and the master's shard broadcast read fields through them afterwards.
The heartbeat count refresh wrote FileCount/DeleteCount into those
records under the lock, which the readers never take, so it was a data
race and a reader could pair a FileCount from one heartbeat with a
DeleteCount from the next. Disk.AddOrUpdateEcShard had the same write
on the incremental path, and its shard-bits/ratio merge was already
in place.
Stored records are now never written after they are published: a count
refresh, an incremental merge and a shard delete each build a copy
beside the stored record and swap the map entry under ecShardsLock, so
a reader holding the old pointer keeps one heartbeat's snapshot. The
refresh compares against whatever record is current under the lock,
since the incremental path may have replaced it since UpdateEcShards
looked. The 0/0-means-not-reported rule is unchanged.
The count tests now assert that the stored pointer changes while a
previously obtained one still reads its old values, and a -race test
runs full and incremental count refreshes against readers; it failed
on the previous code and passes now.
* weed-vector: look for security.toml in -config_dir first
weed processes search the directory -config_dir names before ., so a
cluster keeping security.toml there left the engine in plaintext.
--config-dir (also -config_dir, WEED_VECTOR_CONFIG_DIR) does the same.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* weed-vector: dial the masters the way weed clients do
The master's certificate must chain to grpc.ca and is not checked against
the host dialed, as weed clients verify (advancedtls.CertVerification), so
masters sharing one workload certificate, or dialed by IP, register as
they do from weed. Instead, grpc.client.master_allowed_server_commonNames,
grpc.client.allowed_server_commonNames or grpc.allowed_server_commonNames
(and the wildcard domain likewise) restrict which server identities are
accepted: the common name or a DNS SAN, or an exact URI SAN, as
security.Authenticator.AuthenticateServer does. These used to be refused
at startup; --tls-server-name is gone with the host check.
The TLS runs in a connector that reads the credentials asynchronously at
every dial, so the connect timeout still covers the reads, the TCP connect
and the handshake together. An IPv6 master's brackets are stripped before
the dial.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* test/s3vectors: the hardened leg allowlists server identities for grpc.client
The gateway and the engine then accept only servers named localhost, which
exercises the engine's allowed_server_commonNames check on the master dial.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* weed-vector: parse the refresh interval with time.ParseDuration's arithmetic
The interval was summed in floating-point seconds, so 0.9ns rounded up to
1ns where Go truncates it to 0 and falls back to the 5h default, 0.3s was
off by a nanosecond, and 3000000h was accepted where Go rejects anything
past the longest time.Duration. Each part is now an integer count of
nanoseconds plus the fraction Go adds, with Go's overflow bounds, and the
value is no longer trimmed, as Go does not trim it either.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* weed-vector: test a master dial whose credential read stalls
The bounded-dial tests passed no credentials or a prebuilt endpoint, so
moving the credential read out of the connect timeout would have failed
none of them. The new test points grpc.ca at a FIFO nobody writes.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* weed-vector: dial a closed port in the stalled credential read test
Against a silent listener, a dial that never read the credentials timed
out with the same message, so the test passed without the read. Nothing
listens on the port now, so only the stalled read can time out.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* weed-vector: test that Security::load searches the config dir
Only candidates() was tested, so load dropping config_dir on the way
would have passed.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* weed-vector: expand a leading ~ in -config_dir like weed's ResolvePath
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* weed-vector: hold the port in the stalled credential read test
A released port can be taken before the dial, and a dial that skipped the
read would then time out and pass. A listener that drops every connection
keeps the port and still fails such a dial at once.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
* admin: keep a reconnected plugin worker registered when its old stream unwinds
cleanupSession removed whatever session was registered under the worker
ID, so a worker that reconnected while its previous stream was still
unwinding lost the new session and its registry entry to the old
stream's deferred cleanup. Tie cleanup to the stream that is ending and
publish the new stream before the registry upsert so a lingering
predecessor sees itself superseded.
* admin: fail an in-flight detection when its worker disconnects
RunDetectionWithReport waited only on the caller's context and the
DetectionComplete channel, so a detection whose worker stream ended
mid-request could only finish by timeout. On a customer cluster a
planned restart of ec-vacuum-1 during an ec_vacuum detection left the
default lane in 'detecting' with every execution slot idle until the
3600s detection timeout, blocking every job type in that lane.
Bind the request to the stream it was sent on and fail the detection
as soon as that stream closes, whether by disconnect or by a reconnect
that replaced the session. A liveness tick also fails it once the
registry no longer considers the worker available (heartbeat stale),
sending a cancel in that case because the stream may still be open.
The detector lease is dropped so the next pick starts fresh, and a
'disconnected' detector activity is recorded. WorkerDisconnectForTest
now goes through the real session cleanup.
* admin: retry a scheduled detection once when its detector drops
When a detection fails because its worker disconnected, pick a detector
again and retry once within the same iteration while the job type's
runtime budget allows, instead of giving the whole cycle up. Any other
failure, or a failed retry, ends the run as before and the lane moves
on; the next cycle re-detects anyway.
* admin: register a plugin worker's hello and stream as one
step
WorkerStream published the stream under sessionsMu but ran the registry
upsert after releasing it, so two concurrent streams for one worker ID
could publish their hellos in the opposite order from their sessions:
the registry then described the older hello's capabilities while only
the newer stream was sendable, and detector picks routed on them.
registerSession upserts the hello and installs the stream under one
sessionsMu critical section, and a hello or heartbeat arriving on a
stream that is no longer the registered one is dropped instead of
rewriting the live stream's entry.
* admin: steer a detection retry off the stream that dropped
The retry picked while the lost worker could still be in the registry:
cleanupSession closed done before taking the registry entry down, and a
stream closed by a reconnect keeps its entry on purpose. The lost worker
ranks first on free slots, so the single retry went back to it and
failed on send while a healthy worker sat idle.
The disconnect error now carries the stream it came from and the retry
picks around it; pickDetector only returns workers whose current stream
is open, excluding registry entries whose stream is closed; and
cleanupSession removes the registry entry before closing done, so a
waiter woken by the closure never sees the dead worker as a candidate.
* weed-vector: read the cluster's security.toml
The engine finds security.toml where weed processes do (., ~/.seaweedfs,
/usr/local/etc/seaweedfs, /etc/seaweedfs) with WEED_* environment
variables over it, and builds gRPC client and server credentials by the
rules of security.LoadClientTLS and LoadServerTLS: a dedicated client
certificate is preferred, both halves of a pair must be set, and grpc.ca
is required once either is.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* weed-vector: gRPC TLS to the masters and from the gateway
The masters are dialed with [grpc.client]'s credentials when the cluster
has them, the master's certificate checked against the host dialed or
--tls-server-name. The engine's own listener takes its identity from
[grpc.vector] and then requires a client certificate chaining to grpc.ca,
which is how the gateway, dialing with [grpc.client], reaches it. The
rustls provider is pinned before the first TLS use, as the other binaries
linking both aws-lc-rs and ring do.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* scaffold: document [grpc.vector]
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* test/s3vectors: put every gRPC hop under mutual TLS in the hardened leg
VECTOR_TLS=1 now also writes [grpc.master], [grpc.volume], [grpc.filer],
[grpc.s3], [grpc.client] and [grpc.vector], and probes the master's and
the engine's gRPC ports: plaintext HTTP/2 and a client without a
certificate are refused, the client certificate gets a SETTINGS frame.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* test/s3vectors: give the test volume server 64 volume slots
Every collection grows 7 volumes and test_vector_bucket_encryption opens
two buckets while the previous test's bucket is still being torn down, so
16 slots were marginal: the Go legs failed CreateIndex now and then with
"failed to find writable volumes" and master.log showing "create 7
volume, created 4: Not enough data nodes found!".
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* weed-vector: reload the listener's certificates and enforce allowed_commonNames
The listener now runs the way weed servers run theirs. The certificate, key
and CA files are re-read every WEED_TLS_CERT_REFRESH_INTERVAL (weed's
default of 5h) and on SIGHUP, and new handshakes use the changed files
without a restart; a file that no longer parses is logged and the working
configuration stays. [grpc.vector] allowed_commonNames and
grpc.allowed_wildcard_domain restrict which client certificates are served,
checked against the subject common name as security.Authenticator and the
Rust volume server's CommonNameVerifier do.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* scaffold: allowed_commonNames and the refresh cadence for [grpc.vector]
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* test/s3vectors: the engine's allowlist keeps an outsider certificate out
The hardened leg sets [grpc.vector] allowed_commonNames = "localhost" and
probes the engine with a second certificate the same CA signed for
CN=outsider: the handshake must be refused, while the gateway's certificate
still gets an HTTP/2 SETTINGS frame.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* weed-vector: take the master's TLS name from the parsed endpoint
A master given without a port (https://master.example) had the whole URL
checked against its certificate.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* weed-vector: bound the master TLS handshake
connect_timeout covers only the TCP connect, so a master that accepted the
connection and never answered the handshake held up the other masters.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* weed-vector: look up security.toml keys the way viper does
An empty WEED_* variable no longer hides the file's value, and keys match
regardless of case, as for weed processes.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* weed-vector: refuse master allowlists it cannot enforce
weed checks a dialed server against allowed_server_commonNames and
allowed_server_wildcard_domain on top of grpc.ca. The engine's master dial
accepted every certificate the CA signed even when one was set; it now
fails at startup and names the key.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* test/s3vectors: read the whole frame header in the gRPC TLS probe
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* test/s3vectors: probe the volume, filer and S3 gRPC listeners too
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* weed-vector: read the master dial credentials only when given masters
A standalone engine never dials with grpc.client, so its settings no longer
stop it from starting.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* weed-vector: run each listener TLS handshake on its own task
The accept loop awaited every handshake before accepting the next
connection, so a peer that connected and sent nothing stopped every
gateway connection after it. Handshakes now run concurrently, each under a
10s timeout.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* weed-vector: read the master dial's TLS files at every dial
The client certificate, key and CA were read once at startup, so after the
files rotated a reconnect still presented the old pair. Each dial now reads
them, as the listener re-reads its own.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* weed-vector: read every unit Go durations accept in the refresh interval
WEED_TLS_CERT_REFRESH_INTERVAL values in ns, us or a leading + fell back
to five hours in the engine while weed servers honored them.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* weed-vector: fall back on a refresh interval too long to hold
An oversized WEED_TLS_CERT_REFRESH_INTERVAL panicked the engine at
startup; weed servers reject it and keep the default.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* weed-vector: refuse allowed_commonNames without the listener's certificate
With [grpc.vector] allowed_commonNames set but no cert and key, the
listener served plaintext and checked no client at all.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* weed-vector: read the master dial's credentials inside its connect timeout
The CA, certificate and key were read synchronously before the timeout
started, so a stalled credential path held up every other master.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* weed-vector: let a set key win over an empty spelling of it
With allowed_server_commonNames = "" beside allowed_server_commonnames
= "master", the lookup stopped at the empty exact match and the
allowlist went unnoticed.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
* admin: place uploaded files by their path inside an uploaded folder
The multipart file name keeps only the base name, so the upload API takes
an optional relativePaths field listing each file's path under the target
directory. Paths that are empty, absolute or contain . or .. components
are rejected per file.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* admin: upload a folder from the file browser
The upload modal gets a folder picker, and a folder dropped on the listing
is walked into, so its files upload with their subfolder paths.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* filer UI: upload a folder
An Upload Folder button picks a whole folder, and a folder dropped on the
listing is walked into. Each file is posted to its subfolder under the
current directory, which the filer creates on the first upload into it.
Progress rows are keyed by that path, so equal names in different
subfolders no longer share a row.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* admin: reject an empty relativePaths field instead of ignoring it
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* admin: upload a large selection in batches below the multipart part limit
Go's multipart parser rejects a form of more than 1000 parts, so a folder
of about a thousand files failed as a whole. The upload modal now sends
500 files per request and sums the results; progress covers all batches,
and the button stays disabled until the last one finishes.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* admin: escape file names in upload error messages
A failed upload reports the client-supplied file or folder path, and the
toast renders its message as HTML.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* admin: list failed uploads in the modal and reload the listing when it closes
A partial upload logged its failures only to the console before the page
reloaded, and a batch that failed after earlier ones succeeded left the
listing stale. The failures now stay in the open modal, and closing it
reloads the listing once any file was uploaded.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* admin: cancel the upload when its modal closes
Closing the modal, including with its Cancel button, left the batch loop
running, so the remaining batches were still written. The batch in flight
is now aborted, which also stops the loop, and the listing reloads since
the aborted request may already have stored files.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* admin: count upload progress by files when the selection has no bytes
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* admin: keep the newest upload selection when a folder drop finishes late
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* admin: cap the upload preview at a fixed number of rows
A folder of many thousands of files rendered one preview row per file.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* filer UI: upload a few files at a time
Every selected file started its own request at once, so a large folder
opened thousands of requests together.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* admin: regenerate the gzipped admin.js
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* admin: reload the listing after a batch fails without a result
A batch that fails mid-request may still have stored some of its files,
so closing the modal reloads the listing.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* admin: stop reading a folder drop once a newer selection replaces it
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* admin: skip the read error of a folder drop a newer selection replaced
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* filer UI: share the four-upload limit across selections
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* admin: regenerate the gzipped admin.js
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* admin: set up the file browser handlers once
admin.js already runs setupFileManagerEventHandlers on DOMContentLoaded,
so the file browser page registered every listener twice and each drop
was read twice.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* rust volume: run each gRPC TLS handshake on its own task
The incoming stream awaited every handshake before accepting the next
connection, so a peer that connected to a TLS gRPC port and sent nothing
stopped every gRPC connection after it. Handshakes now run concurrently,
each bounded by grpc-go's default 120s connection timeout, as a Go server's
are.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* rust volume: retry a failed gRPC accept instead of closing the listener
grpc-go backs off and retries temporary accept errors such as EMFILE.
The TLS incoming stream ended on the first accept error, which stopped
the gRPC port for the rest of the process lifetime.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* rust volume: test that the gRPC handshake timeout closes a silent peer
The handshake timeout becomes a parameter so a test can use a short one.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* rust volume: exit cleanly on SIGTERM and SIGINT
The shutdown path raced the HTTP and gRPC tasks with select!, then
awaited both handles again; tokio panics when a JoinHandle is polled after
it completed, so every signal ended in "JoinHandle polled after
completion" with exit status 101 and the volumes never closed. Waiting
for both tasks with join! is enough: both stop on the shutdown signal, and
a gRPC serve error sends it.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* test/volume_server: fail when a stopped server does not exit cleanly
The harness ignored the exit status after SIGINT, which is how the Rust
volume server's shutdown panic went unseen. Every server it stops must
now exit 0 within 10s. The Go volume servers get -preStopSeconds=0, which
the Rust ones already had, so a stop does not spend the default 10s grace
inside that window.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* test/volume_server: let assertArgPair find flags after a bare flag
rustVolumeArgs carries --volume.allowUntrustedRemoteEndpoints without a
value, so pairs no longer sit at even indexes and
TestRustVolumeArgsIncludeReadMode reported --readMode missing although it
was there.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* rust volume: stop both servers when either one exits
A gRPC task that panicked left the HTTP server and heartbeat running
until a signal arrived. Whichever server task ends first now sends the
shutdown and the other is awaited once, matching Go where a serve
failure is fatal.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* rust volume: report an HTTP task panic as a server error
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
* filer.sync: pass the replayed entry down to the chunk retry gates
The fetch, manifest-resolve, and manifest-upload retry gates only received the
replayed version's mtime, which is not enough to order it against the live
source entry. Pass the entry itself; the gates still compare mtime only, so
behavior is unchanged.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* filer.sync: stop chunk retries with the same supersession check the skip uses
The fetch, manifest-resolve, and manifest-upload retry gates stopped early only
for a gone or strictly newer source entry. A transient error therefore kept
retrying a version the source had rewritten under an older mtime until the
read recovered, while a live lookup that lags the replay could stop the
retries on its newer mtime alone. The error handler that runs once a gate stops
already decides with sourceMovedOn, which orders by commit timestamp and
compares chunk sets; the gates now use it too, and hasSourceNewerVersion has no
callers left.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* filer.sync: a lagging live lookup cannot prove a chunk dropped
A peer filer's event reaches subscribers before the source store applies it,
so the live lookup can return the version the replay replaces. That entry
never listed the replay's new chunk, so sourceDroppedChunk read a 404 on it as
an obsolete version and cut the missing-chunk wait short. Like sourceVersion,
it now ignores a live entry committed before the replayed version.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* filer.sync: test that a replicate failure on a superseded version is skipped
Covers onReplicateChunkError with a transient error for a deleted source entry,
a later commit with an older mtime, and an unchanged source, and points the
unconfirmed-supersession test comment at it instead of the helper test.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* filer.sync: test the manifest gate against a missing volume and a lagging lookup
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* filer.sync: a chunk the live source entry dropped is gone whatever its mtime says
sourceDroppedChunk refused to judge a 404'd chunk when the live source entry
carried an older mtime than the replayed version, treating that as a rollback.
Timestamp-preserving writers (cp -p, rsync -a, tar extracts) produce exactly
that order on every rewrite, so the vacuumed chunk sat in the retry loop for
the full missingSourceChunkGrace before being written off. Whether the live
entry still lists the chunk is independent of mtime order.
* filer.sync: treat a rewritten source entry as moved on regardless of its mtime
sourceVersion only compared chunk sets when the live entry was at least as
new as the replayed one, so a live entry stamped with an older mtime but
holding different chunks was never recognized as the current version. The
sink then fetched the replayed chunks, hit 404 on each vacuumed one, and the
same mtime guard in onReplicateChunkError kept the event from being skipped.
Whether the source rewrote the file is a chunk-set question, not an mtime
one; the live version is synced in its place as before.
* filer.sync: apply a substituted live snapshot even under a newer destination mtime
After sourceVersion swapped in the live entry, UpdateEntry re-ran
chooseUpdateAction on it and a destination whose mtime was above the live
entry's read as a late update, so the current source content was never
written. The live snapshot is the source's current state, not a replay, so
mtime order says nothing about it; repair from it unless the destination
already holds it.
* filer.sync: a live entry committed before the replay does not supersede it
Dropping the mtime guard let any chunk-set difference read as a rewrite,
including a live lookup that lags the event: a peer filer's event reaches
subscribers before the local store applies it, so the lookup can still
return the version the replay replaces. sourceVersion then synced that
older version over the destination and consumed the event. The meta-log
commit stamps order the two versions where mtime cannot, so a live entry
committed before the replay proves nothing.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* filer.sync: check commit order before the mtime supersession
A lagging live entry can carry a newer mtime than the replay, as when the
replayed version was stamped back to an older time, so sourceSupersedes
took it as newer and the older version was synced in place of the replay.
Rule out a live entry committed before the replay first.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* rust volume: count sorted-map files and deletes the way Go does
The Rust sorted needle map still used the counter rule Go dropped when
the index loaders were unified: every .idx row, tombstones included,
counted as a file, and each delete added one file and two deletions.
Go counts put rows as files and a delete as one deletion of the live
row's bytes, so the Go and Rust copies of one volume reported different
FileCount and DeleteCount for the same index.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* rust volume: test canDelete volume counters across restarts
Mirrors the Go store test: a canDelete volume reloaded from .sdx must
count its deletes, and the counters it reports must match what the next
reload computes.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* fix(storage): update deletion metrics in SortedFileNeedleMap.Delete (#12363)
* storage: test canDelete volume counters across restarts
A canDelete volume reloaded from .sdx must count its deletes so
volume.deleteEmpty and vacuum see it drain, and the counters it reports
must match what the next reload computes.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* shell: keep fs.mergeVolumes from reverting entries written during the merge
The walk reads a page of entries and moves their chunks one at a time, then
wrote each entry back unconditionally. A file overwritten in between got its
old version back, and the filer deleted the new version's chunks as garbage,
so an acknowledged write was lost without any error.
The update now carries IF_ENTRY_EQUAL over the entry as the walk read it. When
the filer refuses because the entry changed or was deleted, the copies made on
the target volumes are deleted, their plan room is released, and the entry is
skipped; the source needles stay where the filer still points.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* shell: keep fs.distributeChunks from reverting a file overwritten while it runs
The command looks the entry up once, moves its chunks, and then wrote that
entry back unconditionally, so an overwrite during the redistribution was
reverted and the filer deleted the new version's chunks as garbage.
The update now carries IF_ENTRY_EQUAL over the looked-up entry. When the filer
refuses because the file changed or was deleted, the moved copies and any new
manifest chunks are deleted and the command reports it.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* shell: keep s3.versions.audit -heal from clearing a pointer that just moved
The heal cleared the latest-version pointer on the directory entry the walk
read, unconditionally. A version written between the audit and the heal has
already pointed the directory at a live file, and clearing that pointer hid it.
The heal now carries IF_ENTRY_EQUAL over the audited entry and reports a
refusal as a failed heal.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* shell: keep fs.meta.changeVolumeId from reverting entries written meanwhile
The walk rewrote each entry's volume ids and wrote back the copy it had read,
unconditionally, so an overwrite between the read and the write was reverted
and its chunks were deleted as garbage. The update now carries IF_ENTRY_EQUAL
over the entry as read, and a refused entry is skipped and reported.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* shell: keep the target copies another fs.mergeVolumes run committed
moveChunk keeps the key and cookie, so two overlapping merges of the same
source into the same target write the same fid. When one run commits, the
other's update is refused, and deleting its copies deleted the needle the
committed entry now points at.
A refused update now re-reads the entry, resolving its manifests, and deletes
only the copies it no longer references. If the entry cannot be read, the
copies are left for volume.fsck.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* shell: let a later hard-link sibling retry a merge the filer refused
The first sibling of a HardLinkId claims the move for all of them. When its
update is refused, the claim made every later sibling skip, so the shared
chunks stayed on the source volume for the rest of the run. A refused update
now releases the claim, and a sibling listed after the change retries the move
against the entry as it is now.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* shell: delete the manifest blobs a failed fold in fs.distributeChunks saved
When re-manifestizing failed partway, the blobs already uploaded were only
counted in a log line and stayed on their volumes with nothing referencing
them. They have freshly assigned fids, so they are now deleted with the
cookie-checked delete the other orphan cleanups use.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* shell: check the heal clears the pointer file name too
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* shell: require the shell lock for fs.mergeVolumes -apply
moveChunk keeps the key and cookie, so two overlapping runs that move the same
chunk into the same volume write the same fid. Re-reading the entry before
deleting a refused run's copies covers the run that already committed, but not
one that commits after that read. With the lock, two applying runs can no
longer overlap, as volume.fsck and volume.merge already require. A dry run
still works without it.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* shell: check unreferencedTargets reads the refused entry itself
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* shell: check unreferencedTargets keeps a copy a manifest references
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* shell: check unreferencedTargets keeps the manifest copy and skips an unreadable manifest
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* fix(fsck): inspect references to deleted volumes
* volume.fsck: treat every match-all collection filter as global
ALL_COLLECTIONS and EACH_COLLECTION compile to the same match-all
matcher as an empty filter or "*", but the string check treated them
as a restricted scope: a selected missing volume was rejected and
references to master-absent volumes were reported without being purged.
Use the compiled matcher instead.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* volume.fsck: purge an entry whose manifest is on a missing volume
A top-level chunk on a missing volume already proves the entry is
unreadable, whether or not its manifests resolve. Skipping such entries
kept the very case where the manifest needle sat on the retired volume,
and the remaining unresolved manifest then blocked every later orphan
purge. The missing-needle check on live volumes never had this gate, so
both paths now treat these entries the same. The warning now says only
orphan purging is disabled, which is the only thing the counter gates.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* volume.fsck: report a failed Filer delete
A rejected purge DELETE printed nothing unless -v was set, so the run
looked successful after "deleting path" while the entry stayed. The
filer answers 204 for an entry that is already gone, so only real
failures print.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* volume.fsck: tighten help text, comments and missing-volume tests
The help now wraps like the rest of it and says that a purge removes
the whole entry, including its chunks on other volumes. The live test
header no longer calls the run report-only, and the reverse-scan test
checks the report line for each missing chunk.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* volume.fsck: drop the missing-volume design notes
The notes said no destructive change was approved while the change adds
a confirmed purge, still described the dropped manifest gate, and
mostly recorded local runs and a reverse index that is not part of
this change. The operator contract lives in help volume.fsck.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* volume.fsck: allow a report-only missing-volume scan with -collection
The collection check exists to stop a purge that cannot be scoped, but
it also rejected a report-only run that deletes nothing. Apply it only
with -reallyDeleteFilerEntries, and limit the -confirmMissingVolumeDeletion
description to the -volumeId case it guards.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* volume.fsck: count only a 2xx answer as a purged entry
A redirect the client could not follow left the entry in place but was
recorded as deleted, which also queued its parent for empty-directory
cleanup.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* volume.fsck: say that a filtered purge of a missing -volumeId is refused
The help and the early warning both said a purge with -collection only
reports references to unlisted volumes, but a -volumeId naming one is
rejected before the scan. The help now says so, and the warning prints
after that check, so a refused run no longer claims it will report.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: MZ01-CHOIMINSU <mz01-juwannam@MZ01-CHOIMINSU.local>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* admin: scope plugin job and activity fetches to the selected job type
The plugin page fetched the newest 500 jobs and 500 activities
cluster-wide and filtered them by job type in the browser. On a busy
cluster that window can hold none of the selected type (433
erasure_coding + 64 ec_balance jobs, 497 ec_balance activities), so
/plugin/lanes/default/execution?job=ec_vacuum showed "No execution
jobs tracked yet" while the server had seven recent ec_vacuum jobs.
When a job-type tab is active, fetch the queue/detection/execution
lists with job_type=<selected> so the server filters before applying
the limit, and refetch when the selected job type changes. The
unscoped lists are still fetched for the status cards and the overview
per-type summary.
* admin: drop a plugin monitor refresh whose job type is no longer selected
A refresh started on one job-type tab could finish after the user moved
to another tab and that tab's refresh had already landed. Its rows, all
of the earlier job type, then replaced the current tab's and the
renderers filtered them down to empty tables until the next refresh.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* admin: show the lane-wide rows when a job-type scoped fetch fails
On a job-type tab a failed scoped jobs or activities request threw before
anything was stored, so the status cards, scheduler state and tables all
stayed stale. Fall back to the unscoped rows, which the renderers already
filter by job type.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* admin: filter plugin jobs and activities by lane before the limit
/api/plugin/jobs and /api/plugin/activities took the newest N rows across
every lane and then dropped the other lanes' rows, so on a busy default
lane a quieter lane's page could come back empty.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* admin: keep a failed plugin monitor fetch's last rows and report it
Falling back to the lane-wide rows when a scoped fetch failed could still
blank the tables, since that window may hold none of the selected type,
and the failure went unreported. A lane-wide or scheduler-state failure
also threw before the scoped rows that did arrive were stored.
Each request now keeps its previous value on failure: the scoped rows
only while the tab is unchanged, otherwise the lane-wide rows. Whatever
arrived is stored and rendered, then the first failure is rethrown so the
caller reports it.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* admin: report a monitor refresh failure apart from the action it follows
expireJob and runWorkflow refreshed the monitor inside the action's try
block, so a failed refresh after a successful expire or workflow run was
reported as a failed action, and the expired job's detail was not
reloaded. Catch the refresh separately, as runDetection already does.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* admin: show a repeated plugin page refresh failure once
refreshAll runs every 3 seconds and notify opens a modal, so a request
that kept failing reopened the same alert on every tick. Report an error
only when it differs from the last one, and forget it once a refresh
succeeds.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* admin: summarize the EC delete_count accounting warning per refresh
collectCollectionStats runs on every admin collection, bucket and
topology refresh, and it logged a Warning for each EC volume whose
delete_count summed across shard holders exceeds its file_count. That
sum is an expected accounting artefact (a tombstone can be recorded on
more than one holder), it recurs identically on every pass, and on one
cluster with hundreds of such volumes it grew the admin log to 20 GB,
more than 99% of it this one line.
Log one V(1) summary per pass with the affected volume count and keep
the per-volume detail at V(2). The object count is still skipped for
those volumes.
* admin: test that an EC volume with deletes above file_count is skipped
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* test/s3vectors: a hardened cluster, with TLS volume servers, read JWTs and encrypted chunks
Three switches on the test cluster: VECTOR_TLS=1 puts the volume servers
behind HTTPS requiring client certificates and enables https.client for
every process; VECTOR_READ_JWT=1 gives every process a jwt.signing.read.key,
so volume reads need a token; VECTOR_ENCRYPT_VOLUMES=1 starts the gateway
with -encryptVolumeData. The integration matrix gains a hardened leg per
volume server with all three on.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* test/s3vectors: the hardened cluster proves its enforcement
Once the gateway is up, the harness checks that the hardening holds, not
only that it is configured: a TLS volume server refuses plain HTTP (Go
with HTTP 400, Rust with a TLS alert) and a connection without the client
certificate, and answers with it; under a read key an unsigned read of a
real chunk, written through the gateway and located through the filer,
gets 401.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* test/s3vectors: the hardened cluster proves its chunks are encrypted
With VECTOR_ENCRYPT_VOLUMES the probe reads the stored chunk straight
from the volume server, signing the read when a read key is set, and
requires a recorded cipher key and bytes that differ from the plaintext.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* test/s3vectors: the signed probe read must return the chunk
An empty 200 passed both branches; an unencrypted read now has to return
the bytes written, and an encrypted one has to return something else.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* test/s3vectors: a read key in the caller's environment does not leak into the cluster
Only VECTOR_READ_JWT turns read JWTs on; an inherited
WEED_JWT_SIGNING_READ_KEY made the volume servers refuse the probe's
unsigned read in an encrypt-only run.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* test/s3vectors: quote certificate paths as TOML strings in security.toml
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* test/s3vectors: keep non-ASCII certificate paths as UTF-8 in security.toml
TOML has no surrogate-pair escapes, and the Rust volume server reads the
quoted value without decoding escapes at all.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* test/s3vectors: an encrypted probe chunk must not be a readable gzip stream
The gateway may gzip the body before encrypting it, so a chunk uploaded
compressed but unencrypted would differ from the body and still pass.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
* s3api: a known-answer fixture generator for SSE-S3 and filer-encrypted chunks
TestGenerateDirectReadFixture writes, from this package's own encryption,
the pieces a reader outside the gateway must handle: SSE-S3 chunks with
their per-chunk metadata, the object-level key, a single-stream ciphertext,
the KEK derived from s3.sse.key and wrapped under a passphrase in both
on-disk formats, and chunks encrypted by the filer, plain and gzipped. It
is skipped unless DIRECT_READ_FIXTURE names the output file.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* s3api: the fixture generator restores the key manager and checks the write
The test set the global key manager's KEK and left it; a later test in
the same run then found a signing key where it expects none. The previous
key is restored with t.Cleanup. The fixture file's close error is checked
too, so a delayed write failure fails the test instead of leaving a short
fixture.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* s3api: the fixture generator unwraps both KEK formats before writing
The legacy v1 wrap is built by hand in the test; reading v1 and v2 back
through unwrapKEK ties both to the format the key manager accepts.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3api: pin a legacy v1 wrapped KEK made outside the package
Generating the v1 sample through legacyKEKWrappingSalt let a change to
the salt pass on both sides; a vector wrapped elsewhere under the
historical salt must still unwrap.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
* fix(s3api): reject an invalid storage class before uploading the body
* fix(s3api): reject an invalid storage class on CreateMultipartUpload
* fix(s3api): validate storage class and tags before reading the PUT body
* fix(s3api): validate storage class and tags with the other PUT headers
Checking them next to ValidateRequestEncryption keeps the check ahead of
any later hook that may answer the PUT without reaching putToFiler.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* refactor(s3api): reuse validatePutHeaders in putToFiler
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* fix(s3api): store storage class and tags on directory-marker PUTs
The marker path validated x-amz-storage-class and x-amz-tagging but its
mkdir callback never wrote them, so a valid value was accepted and then
dropped and GetObjectTagging on the marker came back empty. Store both
the way putToFiler does.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* test(s3api): read directory-marker tags back through GetObjectTagging
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* filer: say jwt.filer_signing.key gates only the IAM gRPC service
The key never covered the rest of the filer gRPC port, which can read and
write the IAM store under /etc/iam. Point operators at [grpc.filer] mTLS for a
port reachable beyond a trusted network instead of the key.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* filer: note the IAM gRPC OIDC provider and role RPCs are refused without the key
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
At BestSpeed, Go 1.26's flate writes 4000 random bytes plus a 300-byte
zero run as a stored block, so the gzip output was 28 bytes larger than
the input and the test failed its own precondition on the toolchain
go.mod pins. With 450 zero bytes the saving is 5.7% on Go 1.26 and 8.4%
on Go 1.27, inside the marginal range on both.
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* s3tables: resolve s3vectors:VectorBucketTag like s3tables:TableBucketTag
Both name a tag on the bucket that holds the target, whichever resource
the request acts on.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* s3vectors: give vector policy conditions the bucket, index and request tags
A vector bucket policy is evaluated with these condition keys:
- aws:ResourceTag/<key>: the tags of the bucket, or of the index when the
request names one.
- s3vectors:VectorBucketTag/<key>: the bucket's tags, for bucket and
index requests alike.
- aws:RequestTag/<key> and aws:TagKeys: the tags TagResource and
CreateIndex set, and the keys UntagResource removes.
An index's tags are read only when the policy has an aws:ResourceTag
condition. The bucket's come from the lookup that already reads its
policy.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* s3vectors: accept tag conditions in vector bucket policies
PutVectorBucketPolicy rejected every Condition, since no condition key
was supplied. A condition is now accepted when its operator is one the
evaluator knows and each key is aws:ResourceTag/<key>,
aws:RequestTag/<key>, aws:TagKeys or s3vectors:VectorBucketTag/<key>.
Any other key would never match, so it is still rejected, as is a
condition field not spelled Condition.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* test/s3vectors: cover tag conditions in vector bucket policies
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* s3vectors: reject condition values the evaluator cannot compare
A null condition value was accepted and then panicked the request that
matched its statement, a Deny reaching even the bucket owner. A value
must now be a string, number or boolean, or a non-empty list of them.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* admin: carry a file browser upload size limit
The admin server keeps the total size limit of one file browser upload, 500 MB by default as the UI enforced before, and hands it to the file browser page.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* admin: reject file browser uploads over the size limit
The 500 MB limit was only checked in the browser, so any client could post larger uploads and fill the temp dir with spooled parts. Cap the request body at the limit plus room for the multipart framing, and answer 413 when the files add up to more than the limit.
Parse the form before reading the path field. FormValue used to parse it first with the default 32 MB memory, which made the later 1 GB ParseMultipartForm a no-op; keep 32 MB explicitly.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* admin: show and check the configured upload limit in the file browser
The upload dialog reads the limit from the page instead of a hard-coded 500 MB, lists it under the file picker, and skips the check when uploads are unlimited.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* admin: add -uploadMaxSizeMB to set the file browser upload limit
weed admin -uploadMaxSizeMB and weed mini -admin.uploadMaxSizeMB set the total size limit of one file browser upload; 0 removes it.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* admin: note why the upload framing allowance is enough
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* admin: test an upload whose files only exceed the limit together
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* admin: remove spooled upload parts when the upload handler returns
Parts over 32 MB spool to temp files. The router passes the handler a copy of the request, so net/http never sees the parsed form and never removes those files. Every large upload, and every one rejected for size after parsing, left them on disk.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* s3vectors: run an operation for trusted local tooling
Like s3tables.Manager.Execute, Server.Execute runs one S3 Vectors operation
through its handler as a given identity, without a signed request. The admin
console can then manage vector buckets through the same validation,
authorization and catalog steps as the API.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* admin: list, inspect, create and delete vector buckets over the API
/api/s3vectors runs the S3 Vectors operations through Server.Execute as the
admin account: list vector buckets with their encryption, read one bucket
with its policy and each index's data type, dimension, distance metric and
non-filterable keys, create and delete a bucket, and delete an index. Going
through the operations keeps the API's rules, such as ICEBERG buckets not
being vector buckets and the implicit index namespace not blocking a delete.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* s3vectors: export the rule for which table buckets are vector buckets
IsVectorBucketFormat names the check ListVectorBuckets and isVectorBucket
already make, so the admin console can tell the same buckets apart.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* admin: vector bucket pages
Object Store > Vector Buckets lists the vector buckets with their ARN,
encryption and creation time, creates and deletes them, and opens one bucket
with its indexes (data type, dimension, distance metric, non-filterable keys)
and its policy, where an index can be deleted. A LANCE row on the Table
Buckets page links to the same bucket's vector page, since both APIs serve it.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* admin: stop the vector bucket and index lists at the list limit
A page cut short by filtered ICEBERG buckets let the next page carry
the list past s3TablesAdminListLimit.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* admin: say when a vector bucket or index list is cut at the limit
The lists stop at s3TablesAdminListLimit while more remain, so mark
them truncated and point to an S3 Vectors client for the rest.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* admin: indent the vector bucket policy
A policy stored as one line ran off the right edge of the details page.
Policies that are not valid JSON are shown as stored.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* admin: show a deeply nested vector bucket policy as stored
Indenting multiplies the size of a deeply nested document, so a policy
past 32 levels of nesting is shown without indenting.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* volume(rust): replicate the stored needle instead of the request body
The primary forwarded the original request body and headers, so a
replica re-derived compression from the raw bytes. Send what Go's
ReplicatedWrite sends: a one-part multipart upload of the stored bytes,
gzip-marked when compressed, with the name, mime, pairs and Content-MD5,
and carry ts, ttl, cm and fsync in the query so form-field values reach
the replica too.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* volume(rust): keep the primary's compression on replica writes
A type=replicate upload now stores the bytes as received, like Go's
ParseUpload, so a replica no longer compresses data the primary stored
uncompressed.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* volume(rust): replicas keep the primary's name and mime
Forward the primary's filename and mime even past the 255-byte needle
fields, so a replica applies the same limit and sets the same flags,
and skip the filename MIME guess on type=replicate, so an untyped
upload stays untyped on every copy.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* volume(rust): keep the multipart part Content-Type as sent
Parsing the part type as a mime value lowercased it and dropped values
it could not parse, so a replica stored a different mime than the
primary. Read the header verbatim, as Go's part.Header.Get does.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* volume: replicas keep the primary's mime as forwarded
A Go replica guessed the mime from the file extension, so an upload the
primary stored as application/octet-stream, or with no mime at all, got
a different type on the replica. On type=replicate keep the part's
Content-Type as sent, and have the replication sender leave an empty
mime empty instead of filling it in from the extension. This matches
what the Rust replica does.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* volume(rust): pick the JPEG orientation fix by file name, as Go does
Go checks the mime and the upload's file name, Rust checked the mime and
the URL path, so the same upload could be rotated on one server and not
on the other. With the name and mime forwarded as stored, a replica now
reaches the primary's decision, and a rotated image has no EXIF tag left
to act on, so the fix stays a no-op there.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* s3vectors: tag vector buckets and indexes
TagResource, UntagResource and ListTagsForResource answer on
/tags/{resourceArn} (POST, DELETE, GET) and by name. Each authorizes its
own s3vectors action on the bucket or index ARN, then reads or writes the
catalog record's tags. A resource holds at most 50 tags.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3vectors: apply the tags passed to CreateVectorBucket and CreateIndex
The tags field was accepted and dropped. It is now stored on the new
bucket or index, and creating with tags needs s3vectors:TagResource in
addition to the create action.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* test/s3vectors: cover resource tagging end to end
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3tables: keep tag operations to a bound table bucket
TagResource, UntagResource and ListTagsForResource run through Execute
with WithTableBucketCreatedAt now answer not found when the bucket name
holds another bucket, as the bound creates and deletes already do.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3vectors: bind tagging to the authorized bucket and to vector indexes
The catalog tag calls carry the creation time of the bucket authorizeVector
approved, so a bucket created again under the name is not touched. An index
ARN must name a Lance table: another table in the vector namespace of a
format-less bucket is refused, as DeleteIndex does.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3vectors: match a TagResource grant scoped to the bucket being created
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3tables: merge tag writes into the tags the record holds at the write
TagResource and UntagResource merged into the tags they read before the
compare-and-swap and wrote that map on every attempt, so a tag write that
landed in between was undone. The merge now runs inside the mutation.
WithMaxTags lets an in-process caller cap the merged set there.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3vectors: check the 50-tag limit inside the catalog's tag write
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3tables: check a tag operation's bound bucket after its reads
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3vectors: validate each tag key UntagResource removes
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3vectors: validate untag keys one at a time
UntagResource sized a map by the number of submitted tag keys before
validating them, so a large body of repeated keys allocated far more than
the request itself, before authorization ran.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3vectors: authorize UntagResource before validating its keys
Validating a near-32 MiB body of tag keys costs about 13x its JSON
decode, so an unauthorized caller could make the gateway spend that
work on a request it then refuses.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* filer: test the rename subdirectory check through CanRename
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* filer: reuse IsEqualOrUnder for the rename subdirectory check
A trailing slash on the old name made the source path end in a slash,
so a child of the source was no longer seen as under it.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* compression: compress JSON uploads, and match file types by extension
IsCompressableFileType did not recognise JSON media types, so S3 uploads of application/json (which carry no file name) were never compressed. Its callers also passed filepath.Base of the file name where it expects the extension, so the .json/.js/.html/... extension rules never matched a real file name. Both callers now pass filepath.Ext; the function lower-cases the extension and media type, ignores media-type parameters ("application/json; charset=utf-8"), and treats */json and *+json types as compressible.
* compression: treat application/json-seq as JSON
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* volume(rust): compress JSON uploads and match media types without parameters
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* operation: keep uploader gzip only when it saves at least 10%
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Pat White <5015863+patwhite@users.noreply.github.com>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
CanRename rejected a move whenever the new parent directory's path started
with the source path as a plain string, so moving /dir/a into /dir/a.d
(or anything under /dir/ab) failed with "can not move directory to a
subdirectory of itself". Check the path boundary instead: reject only
when the new parent is the source itself or lies under it.
Claude-Session: https://claude.ai/code/session_01VncHKTehVroSAzRMUHykEX
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* s3tables: bind DeleteNamespace to the authorized namespace
The delete claimed whatever finished namespace held the name at claim
time, so one deleted and created again by another owner after the
authorization read was removed under the first one's authorization.
The claim now requires the creation time that was authorized.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3vectors: bind DeleteVectorBucket's namespace cleanup to the authorized bucket
After the bound DeleteTableBucket reported the bucket not empty, the
index namespace cleanup ran unbound, so a bucket deleted and created
again in between lost its empty index namespace under the first
bucket's authorization. DeleteNamespace now honors the bucket binding,
checking the bucket after reading the namespace, and the cleanup
carries it.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-02 09:55:09 -08:00
811 changed files with 84231 additions and 8358 deletions
@@ -17,7 +17,9 @@ SeaweedFS is a simple and highly scalable distributed file system. There are two
1. to store billions of files!
2. to serve the files fast!
One `weed` binary serves an S3 object store, a POSIX file system, and a lakehouse with S3 Tables, all over the same data. Each blob is one disk read away, capacity grows by starting another volume server, and cloud storage can be cached or tiered transparently. Both read and write operations have O(1) complexity and can run at the full speed supported by the underlying hardware.
One SeaweedFS cluster serves an S3 object store, a POSIX file system, a lakehouse with S3 Tables and a built-in Iceberg REST Catalog, and [vector search with S3 Vector Buckets](#vector-search-with-s3-vector-buckets), all on shared storage. Each blob is one disk read away, capacity grows by starting another volume server, and cloud storage can be cached or tiered transparently. Blob reads and writes have O(1) complexity and can run at the full speed supported by the underlying hardware.

- [Download Binaries for different platforms](https://github.com/seaweedfs/seaweedfs/releases/latest)
* [The most complete S3 API](#the-most-complete-s3-api)
* [A data warehouse with S3 Tables](#a-data-warehouse-with-s3-tables)
* [Vector search with S3 Vector Buckets](#vector-search-with-s3-vector-buckets)
* [A fast cache for cloud storage](#a-fast-cache-for-cloud-storage)
* [Active-active replication and more](#active-active-replication-and-more)
* [Architecture](#architecture)
@@ -204,12 +207,13 @@ On one laptop, [`weed benchmark`][Benchmarks] writes 1KB files at 15,700 per sec
## The most complete S3 API ##
The S3 gateway implements the object, bucket, S3 Tables, IAM, and STS APIs on one endpoint, so the AWS SDKs and CLI, rclone, restic, Spark, and Trino work unchanged.
The S3 gateway implements the object, bucket, S3 Tables, S3 Vectors, IAM, and STS APIs on one endpoint, so the AWS SDKs and CLI, rclone, restic, Spark, and Trino work unchanged. S3 Vectors uses the `weed-vector` service described [below](#vector-search-with-s3-vector-buckets).
| API | Operations |
| --- | --- |
| S3 bucket and object | 73 |
| S3 Tables | 36 |
| S3 Vectors | 19 |
| IAM | 39 |
| STS | 5 |
@@ -223,7 +227,7 @@ The full operation list is in [Amazon S3 API][AmazonS3API], and [Supported APIs
## A data warehouse with S3 Tables ##
SeaweedFS is a lakehouse in one system. [S3 Table Buckets][S3TableBucket] hold Apache Iceberg tables by default, or [Lance][LanceCatalog] tables for vectors and multimodal data, and the built-in [Iceberg REST Catalog][IcebergCatalog] and Lance namespace serve them directly. There is no Hive Metastore, Glue, or separate catalog service to deploy, secure, and back up.
SeaweedFS is a lakehouse in one system. [S3 Table Buckets][S3TableBucket] hold Apache Iceberg tables by default, or [Lance][LanceCatalog] tables for vectors and multimodal data, and the built-in [Iceberg REST Catalog][IcebergCatalog] and Lance namespace serve them directly. The S3 Tables API and Iceberg REST Catalog share the same namespaces, table metadata, and permissions. There is no Hive Metastore, Glue, or separate catalog service to deploy, secure, and back up.
* Query engines operate on the same tables at the same time: [Spark][SparkIceberg], [Trino][TrinoIceberg], [Dremio][DremioIceberg], [DuckDB][DuckDBIceberg], [Apache Doris][DorisIceberg], [RisingWave][RisingWaveIceberg], ClickHouse, and [LanceDB][LanceDB]. Catalog commits are atomic compare-and-swap, so concurrent writers are safe. [Lakekeeper][Lakekeeper] can front the same storage with STS-vended credentials. [OLake Go][OLake] is a high-performance, open-source EL (Extract–Load) platform that bridges operational databases and open lakehouse storage - writing Apache Iceberg tables or Parquet files on Amazon S3.
* [Automated table maintenance][IcebergMaintenance]: compaction, snapshot expiration, orphan file removal, and manifest rewriting, configured per bucket or table through the S3 Tables maintenance APIs, and the same for [Lance][LanceMaintenance].
@@ -232,6 +236,40 @@ SeaweedFS is a lakehouse in one system. [S3 Table Buckets][S3TableBucket] hold A
`S3_TABLE_BUCKET=warehouse ./weed mini -dir=./data` brings the whole stack up on a laptop.
## Vector search with S3 Vector Buckets ##
Keep embeddings alongside your objects and tables in the same SeaweedFS cluster. S3 Vector Buckets expose the AWS S3 Vectors API for semantic search, retrieval-augmented generation (RAG), and recommendations, using Lance tables on SeaweedFS storage.
* Create vector buckets and indexes; insert, retrieve, list, and delete vectors with the AWS SDKs or CLI pointed at your S3 endpoint.
* Find the nearest vectors with cosine or Euclidean distance, filter by metadata, and return distances and metadata with the results.
* Control access with `s3vectors:*` IAM actions and vector bucket policies; tag buckets and indexes for organization and access control.
Install the separate vector engine; the default installer installs only `weed`:
Release 4.53 provides vector binaries for Linux amd64 and arm64. If your release has no binary for your platform, build from [source](seaweed-vector). From a directory where you want a new checkout:
If you already have a checkout, start at its root and skip the clone and `cd seaweedfs` steps.
For Windows source builds, set `PROTOC` to a native `protoc.exe` before running Cargo; the [desktop build workflow](.github/workflows/rust-vector-platforms.yml) shows how to select the vendored executable.
Run the installed `weed-vector` binary alongside the local cluster started with `weed mini` above (or use `seaweed-vector/target/release/weed-vector` after building from source):
```bash
weed-vector --masters=localhost:9333
```
It registers with the master and the S3 gateway discovers it automatically. An AWS S3 Vectors client uses the same S3 endpoint at http://localhost:8333. The vector engine stores its data through the cluster's filers and volume servers, so embeddings share the storage layer's replication and erasure coding. See the [AWS SDK examples](test/s3vectors/s3vectors_test.py) for bucket, index, and query operations.
## A fast cache for cloud storage ##
[Cloud Drive][CloudDrive] mounts a bucket from S3, Google Cloud Storage, Azure, Backblaze B2, Wasabi, Storj, or any S3-compatible store into SeaweedFS and serves it at local speed:
The S3 gateway serves the S3 Vectors API on its existing endpoint. Enable a vector engine and Lance maintenance worker alongside the cluster:
```yaml
s3:
enabled: true
vector:
enabled: true
replicas: 1
worker:
enabled: true
admin:
enabled: true
secret:
adminPassword: change-me
global:
seaweedfs:
securityConfig:
jwtSigning:
filerWrite: true
```
The engine registers its pod address with the masters and stores indexes through the filers and volume servers. Both Go and Rust volume servers use the same setup. The maintenance worker builds and updates ANN indexes; queries can scan data before indexing completes. Enabling filer write signing also requires clients writing directly to the filer HTTP API to supply a signed JWT.
The bundled engine images support Linux amd64 and arm64. Use `vector.imageOverride` to select a compatible engine build, `vector.masters` for an external cluster, and `vector.resources` to set memory and CPU limits. The cache defaults to 512 MiB and is configured with `vector.readCacheMB`.
`global.seaweedfs.enableSecurity` provisions and mounts the vector certificate and shared security configuration. With external certificates, provide the `<fullname>-vector-cert` secret (`my-seaweedfs-vector-cert` for release `my`). `fullname` follows `seaweedfs.fullname`, including `fullnameOverride`, `nameOverride`, and DNS-name truncation. Provide this secret alongside the existing CA and client secrets.
The default `vector.metricsPort: 0` and TCP probes support existing engine images. The default 4.53 image does not accept metrics flags; enabling metrics with it is rejected at render time. For engine builds with `/metrics`, `/health`, and `/ready`, set `vector.imageOverride` to that compatible image and `vector.metricsPort: 9329`; readiness then checks master registration. Enabling `global.seaweedfs.monitoring.enabled` also creates the vector ServiceMonitor. NetworkPolicy includes the configured gRPC and metrics ports.
To provision vector buckets during installation and upgrades, set:
```yaml
vector:
enabled: true
createBuckets:
- embeddings
```
The hook uses the AWS CLI's S3 Vectors API and works with existing SeaweedFS vector releases. It creates missing buckets, preserves existing vector buckets, and fails if the name belongs to an Iceberg bucket. It uses the chart's generated S3 admin credentials, including `s3.credentials.admin.existingSecret`. With a custom `s3.existingConfigSecret` or an external endpoint, set `vector.bucketHook.credentialsSecret` to a Secret containing `AWS_ACCESS_KEY_ID` and `AWS_SECRET_ACCESS_KEY` (and optionally `AWS_SESSION_TOKEN`). Key names are configurable. `vector.bucketHook.endpoint` overrides the internal S3 service URL; `caBundleSecret` and `caBundleKey` configure a private HTTPS CA. The hook image and resource limits are configurable under `vector.bucketHook`. For restricted Pod Security, select a non-root-compatible image and configure `vector.bucketHook.podSecurityContext` and `containerSecurityContext`, including `runAsNonRoot` and the image user ID.
"description":"Vector engine metrics require HTTP metrics to be enabled. Gateway bucket labels require vector bucket attribution.",
"fieldConfig":{
"defaults":{
"color":{
"mode":"palette-classic"
},
"custom":{
"axisBorderShow":false,
"axisCenteredZero":false,
"axisColorMode":"text",
"axisLabel":"",
"axisPlacement":"auto",
"barAlignment":0,
"drawStyle":"line",
"fillOpacity":0,
"gradientMode":"none",
"hideFrom":{
"legend":false,
"tooltip":false,
"viz":false
},
"insertNulls":false,
"lineInterpolation":"linear",
"lineWidth":1,
"pointSize":4,
"scaleDistribution":{
"type":"linear"
},
"showPoints":"auto",
"spanNulls":false,
"stacking":{
"group":"A",
"mode":"none"
},
"thresholdsStyle":{
"mode":"off"
}
},
"mappings":[],
"thresholds":{
"mode":"absolute",
"steps":[
{
"color":"green",
"value":null
}
]
},
"unit":"reqps",
"unitScale":true
},
"overrides":[]
},
"gridPos":{
"h":8,
"w":12,
"x":0,
"y":140
},
"id":96,
"links":[],
"options":{
"legend":{
"calcs":[],
"displayMode":"list",
"placement":"bottom",
"showLegend":true
},
"tooltip":{
"mode":"single",
"sort":"none"
}
},
"pluginVersion":"8.1.2",
"targets":[
{
"datasource":{
"type":"prometheus",
"uid":"${DS_PROMETHEUS}"
},
"editorMode":"code",
"expr":"sum by (type, bucket) (rate(SeaweedFS_s3_request_total{namespace=~\"$NAMESPACE\",type=~\"s3vectors\\\\..*\"}[$__rate_interval]))",
"range":true,
"refId":"A",
"legendFormat":"{{type}} {{bucket}}"
}
],
"title":"Vector API requests by bucket",
"type":"timeseries"
},
{
"datasource":{
"type":"prometheus",
"uid":"${DS_PROMETHEUS}"
},
"description":"Vector engine metrics require HTTP metrics to be enabled. Gateway bucket labels require vector bucket attribution.",
"fieldConfig":{
"defaults":{
"color":{
"mode":"palette-classic"
},
"custom":{
"axisBorderShow":false,
"axisCenteredZero":false,
"axisColorMode":"text",
"axisLabel":"",
"axisPlacement":"auto",
"barAlignment":0,
"drawStyle":"line",
"fillOpacity":0,
"gradientMode":"none",
"hideFrom":{
"legend":false,
"tooltip":false,
"viz":false
},
"insertNulls":false,
"lineInterpolation":"linear",
"lineWidth":1,
"pointSize":4,
"scaleDistribution":{
"type":"linear"
},
"showPoints":"auto",
"spanNulls":false,
"stacking":{
"group":"A",
"mode":"none"
},
"thresholdsStyle":{
"mode":"off"
}
},
"mappings":[],
"thresholds":{
"mode":"absolute",
"steps":[
{
"color":"green",
"value":null
}
]
},
"unit":"s",
"unitScale":true
},
"overrides":[]
},
"gridPos":{
"h":8,
"w":12,
"x":12,
"y":140
},
"id":97,
"links":[],
"options":{
"legend":{
"calcs":[],
"displayMode":"list",
"placement":"bottom",
"showLegend":true
},
"tooltip":{
"mode":"single",
"sort":"none"
}
},
"pluginVersion":"8.1.2",
"targets":[
{
"datasource":{
"type":"prometheus",
"uid":"${DS_PROMETHEUS}"
},
"editorMode":"code",
"expr":"histogram_quantile(0.95, sum by (le, bucket) (rate(SeaweedFS_s3_request_seconds_bucket{namespace=~\"$NAMESPACE\",type=\"s3vectors.QueryVectors\"}[$__rate_interval])))",
"range":true,
"refId":"A",
"legendFormat":"{{bucket}}"
}
],
"title":"Vector query p95 latency",
"type":"timeseries"
},
{
"datasource":{
"type":"prometheus",
"uid":"${DS_PROMETHEUS}"
},
"description":"Vector engine metrics require HTTP metrics to be enabled. Gateway bucket labels require vector bucket attribution.",
"fieldConfig":{
"defaults":{
"color":{
"mode":"palette-classic"
},
"custom":{
"axisBorderShow":false,
"axisCenteredZero":false,
"axisColorMode":"text",
"axisLabel":"",
"axisPlacement":"auto",
"barAlignment":0,
"drawStyle":"line",
"fillOpacity":0,
"gradientMode":"none",
"hideFrom":{
"legend":false,
"tooltip":false,
"viz":false
},
"insertNulls":false,
"lineInterpolation":"linear",
"lineWidth":1,
"pointSize":4,
"scaleDistribution":{
"type":"linear"
},
"showPoints":"auto",
"spanNulls":false,
"stacking":{
"group":"A",
"mode":"none"
},
"thresholdsStyle":{
"mode":"off"
}
},
"mappings":[],
"thresholds":{
"mode":"absolute",
"steps":[
{
"color":"green",
"value":null
}
]
},
"unit":"reqps",
"unitScale":true
},
"overrides":[]
},
"gridPos":{
"h":8,
"w":12,
"x":0,
"y":148
},
"id":98,
"links":[],
"options":{
"legend":{
"calcs":[],
"displayMode":"list",
"placement":"bottom",
"showLegend":true
},
"tooltip":{
"mode":"single",
"sort":"none"
}
},
"pluginVersion":"8.1.2",
"targets":[
{
"datasource":{
"type":"prometheus",
"uid":"${DS_PROMETHEUS}"
},
"editorMode":"code",
"expr":"sum by (type, code) (rate(SeaweedFS_s3_request_total{namespace=~\"$NAMESPACE\",type=~\"s3vectors\\\\..*\",code=~\"[45]..\"}[$__rate_interval]))",
"range":true,
"refId":"A",
"legendFormat":"{{type}} {{code}}"
}
],
"title":"Vector API errors",
"type":"timeseries"
},
{
"datasource":{
"type":"prometheus",
"uid":"${DS_PROMETHEUS}"
},
"description":"Vector engine metrics require HTTP metrics to be enabled. Gateway bucket labels require vector bucket attribution.",
"fieldConfig":{
"defaults":{
"color":{
"mode":"palette-classic"
},
"custom":{
"axisBorderShow":false,
"axisCenteredZero":false,
"axisColorMode":"text",
"axisLabel":"",
"axisPlacement":"auto",
"barAlignment":0,
"drawStyle":"line",
"fillOpacity":0,
"gradientMode":"none",
"hideFrom":{
"legend":false,
"tooltip":false,
"viz":false
},
"insertNulls":false,
"lineInterpolation":"linear",
"lineWidth":1,
"pointSize":4,
"scaleDistribution":{
"type":"linear"
},
"showPoints":"auto",
"spanNulls":false,
"stacking":{
"group":"A",
"mode":"none"
},
"thresholdsStyle":{
"mode":"off"
}
},
"mappings":[],
"thresholds":{
"mode":"absolute",
"steps":[
{
"color":"green",
"value":null
}
]
},
"unit":"short",
"unitScale":true
},
"overrides":[]
},
"gridPos":{
"h":8,
"w":12,
"x":12,
"y":148
},
"id":99,
"links":[],
"options":{
"legend":{
"calcs":[],
"displayMode":"list",
"placement":"bottom",
"showLegend":true
},
"tooltip":{
"mode":"single",
"sort":"none"
}
},
"pluginVersion":"8.1.2",
"targets":[
{
"datasource":{
"type":"prometheus",
"uid":"${DS_PROMETHEUS}"
},
"editorMode":"code",
"expr":"last_over_time(SeaweedFS_vector_connected{namespace=~\"$NAMESPACE\"}[5m]) * on (cluster, namespace, instance, job) up{namespace=~\"$NAMESPACE\"}",
"range":true,
"refId":"A",
"legendFormat":"{{instance}}"
}
],
"title":"Registered vector engines",
"type":"timeseries"
},
{
"datasource":{
"type":"prometheus",
"uid":"${DS_PROMETHEUS}"
},
"description":"Vector engine metrics require HTTP metrics to be enabled. Gateway bucket labels require vector bucket attribution.",
"fieldConfig":{
"defaults":{
"color":{
"mode":"palette-classic"
},
"custom":{
"axisBorderShow":false,
"axisCenteredZero":false,
"axisColorMode":"text",
"axisLabel":"",
"axisPlacement":"auto",
"barAlignment":0,
"drawStyle":"line",
"fillOpacity":0,
"gradientMode":"none",
"hideFrom":{
"legend":false,
"tooltip":false,
"viz":false
},
"insertNulls":false,
"lineInterpolation":"linear",
"lineWidth":1,
"pointSize":4,
"scaleDistribution":{
"type":"linear"
},
"showPoints":"auto",
"spanNulls":false,
"stacking":{
"group":"A",
"mode":"none"
},
"thresholdsStyle":{
"mode":"off"
}
},
"mappings":[],
"thresholds":{
"mode":"absolute",
"steps":[
{
"color":"green",
"value":null
}
]
},
"unit":"reqps",
"unitScale":true
},
"overrides":[]
},
"gridPos":{
"h":8,
"w":12,
"x":0,
"y":156
},
"id":100,
"links":[],
"options":{
"legend":{
"calcs":[],
"displayMode":"list",
"placement":"bottom",
"showLegend":true
},
"tooltip":{
"mode":"single",
"sort":"none"
}
},
"pluginVersion":"8.1.2",
"targets":[
{
"datasource":{
"type":"prometheus",
"uid":"${DS_PROMETHEUS}"
},
"editorMode":"code",
"expr":"sum by (type, code) (rate(SeaweedFS_vector_request_total{namespace=~\"$NAMESPACE\",code!=\"OK\"}[$__rate_interval]))",
"range":true,
"refId":"A",
"legendFormat":"{{type}} {{code}}"
}
],
"title":"Vector engine request failures",
"type":"timeseries"
},
{
"datasource":{
"type":"prometheus",
"uid":"${DS_PROMETHEUS}"
},
"description":"Vector engine metrics require HTTP metrics to be enabled. Gateway bucket labels require vector bucket attribution.",
"fieldConfig":{
"defaults":{
"color":{
"mode":"palette-classic"
},
"custom":{
"axisBorderShow":false,
"axisCenteredZero":false,
"axisColorMode":"text",
"axisLabel":"",
"axisPlacement":"auto",
"barAlignment":0,
"drawStyle":"line",
"fillOpacity":0,
"gradientMode":"none",
"hideFrom":{
"legend":false,
"tooltip":false,
"viz":false
},
"insertNulls":false,
"lineInterpolation":"linear",
"lineWidth":1,
"pointSize":4,
"scaleDistribution":{
"type":"linear"
},
"showPoints":"auto",
"spanNulls":false,
"stacking":{
"group":"A",
"mode":"none"
},
"thresholdsStyle":{
"mode":"off"
}
},
"mappings":[],
"thresholds":{
"mode":"absolute",
"steps":[
{
"color":"green",
"value":null
}
]
},
"unit":"short",
"unitScale":true
},
"overrides":[]
},
"gridPos":{
"h":8,
"w":12,
"x":12,
"y":156
},
"id":101,
"links":[],
"options":{
"legend":{
"calcs":[],
"displayMode":"list",
"placement":"bottom",
"showLegend":true
},
"tooltip":{
"mode":"single",
"sort":"none"
}
},
"pluginVersion":"8.1.2",
"targets":[
{
"datasource":{
"type":"prometheus",
"uid":"${DS_PROMETHEUS}"
},
"editorMode":"code",
"expr":"sum by (type) (SeaweedFS_vector_in_flight_requests{namespace=~\"$NAMESPACE\"})",
"range":true,
"refId":"A",
"legendFormat":"{{type}}"
}
],
"title":"Vector engine requests in flight",
"type":"timeseries"
},
{
"datasource":{
"type":"prometheus",
"uid":"${DS_PROMETHEUS}"
},
"description":"Vector engine metrics require HTTP metrics to be enabled. Gateway bucket labels require vector bucket attribution.",
"fieldConfig":{
"defaults":{
"color":{
"mode":"palette-classic"
},
"custom":{
"axisBorderShow":false,
"axisCenteredZero":false,
"axisColorMode":"text",
"axisLabel":"",
"axisPlacement":"auto",
"barAlignment":0,
"drawStyle":"line",
"fillOpacity":0,
"gradientMode":"none",
"hideFrom":{
"legend":false,
"tooltip":false,
"viz":false
},
"insertNulls":false,
"lineInterpolation":"linear",
"lineWidth":1,
"pointSize":4,
"scaleDistribution":{
"type":"linear"
},
"showPoints":"auto",
"spanNulls":false,
"stacking":{
"group":"A",
"mode":"none"
},
"thresholdsStyle":{
"mode":"off"
}
},
"mappings":[],
"thresholds":{
"mode":"absolute",
"steps":[
{
"color":"green",
"value":null
}
]
},
"unit":"percentunit",
"unitScale":true
},
"overrides":[]
},
"gridPos":{
"h":8,
"w":12,
"x":0,
"y":164
},
"id":102,
"links":[],
"options":{
"legend":{
"calcs":[],
"displayMode":"list",
"placement":"bottom",
"showLegend":true
},
"tooltip":{
"mode":"single",
"sort":"none"
}
},
"pluginVersion":"8.1.2",
"targets":[
{
"datasource":{
"type":"prometheus",
"uid":"${DS_PROMETHEUS}"
},
"editorMode":"code",
"expr":"(sum by (cluster, namespace, job, instance) (rate(SeaweedFS_vector_cache_requests_total{namespace=~\"$NAMESPACE\",result=\"hit\"}[$__rate_interval])) or on (cluster, namespace, job, instance) (0 * sum by (cluster, namespace, job, instance) (rate(SeaweedFS_vector_cache_requests_total{namespace=~\"$NAMESPACE\"}[$__rate_interval])))) / sum by (cluster, namespace, job, instance) (rate(SeaweedFS_vector_cache_requests_total{namespace=~\"$NAMESPACE\"}[$__rate_interval]))",
{{- if and .Values.vector.metricsPort (not .Values.vector.imageOverride) (eq (toString (.Values.image.tag | default .Chart.AppVersion)) "4.53") }}
{{- fail "vector.metricsPort requires vector.imageOverride selecting an engine with metrics support; the default 4.53 image does not support metrics" }}
<titleid="title">SeaweedFS: one cluster for objects, files, tables, and vectors</title>
<descid="description">S3 object storage, a POSIX file system, table buckets with a built-in Iceberg REST Catalog and Lance namespace, and S3 Vector Buckets share one SeaweedFS storage cluster. Masters coordinate the cluster, filers manage namespaces and metadata, and volume servers store replicated or erasure-coded blobs. Cloud caching, cloud tiering, and cross-cluster replication extend the cluster.</desc>
`weed-vector` serves the S3 Vectors data plane through the S3 gateway. Indexes use Lance tables on the cluster's filers and volume servers. Go and Rust volume servers use the same engine and API.
Point an AWS S3 Vectors client at the S3 endpoint, normally `http://localhost:8333`. The engine registers with the masters automatically. `--listen` defaults to `127.0.0.1:8090`; use an address reachable from the gateway when running on another host or in a container.
The HTTP listener provides `/health`, `/ready`, and `/metrics`. Readiness requires an active registration with the master. The gRPC listener also implements the standard health service for `vector_pb.VectorService` and the whole server. These health statuses follow master registration.
Metrics use the `SeaweedFS_vector_` prefix:
-`connected`: active master registration.
-`request_total{type,code}`, `request_seconds{type}`, and `in_flight_requests{type}`: engine requests and their gRPC outcomes.
-`query_results_total`: vectors returned by queries.
-`cache_requests_total{result}`, `cache_bytes`, and `cache_entries`: immutable read cache activity and occupancy.
-`storage_read_bytes_total` and `storage_read_seconds`: storage fetches for cacheable immutable reads.
The S3 gateway separately publishes `SeaweedFS_s3_` request metrics with `type="s3vectors.<Operation>"`. Scrape both processes to distinguish gateway latency from engine work. Metrics and HTTP probes are unauthenticated; bind them to an internal address or restrict their network access.
For mutual TLS, configure `[grpc.vector]` for the engine's server certificate, `[grpc.client]` for its master/filer connections, and the cluster CA. HTTPS volume reads also use `[https.client]`. The engine follows the same signing and encryption configuration as the other cluster clients.
Run the Lance maintenance worker against the same masters and Lance namespace to build and update ANN indexes. Queries can scan data that has not been indexed yet. See the [maintenance worker documentation](../seaweed-worker/README.md) and [Helm chart](../k8s/charts/seaweedfs/README.md) for deployment settings.
The integration suite at `test/s3vectors` runs the same API and observability checks against both Go and Rust volume servers, including mutual TLS, signed volume reads, and encrypted volume data.
"Reads of an EC interval served by reconstructing from other shards; success means reconstruction produced data, not that the needle passed validation.",
),
&["result"],
)
.expect("metric can be created")
});
/// Counter of storage read/write EIO errors on volumes and EC shards.
Loaded 100 of 811 files, more files were not shown because too many files have changed in this diff.
Show more
Reference in new issue
Block a user
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.