Commit Graph
16301 Commits
Author SHA1 Message Date
Chris LuandClaude Opus 5.5 04c7b12238 s3api: reject unknown notification elements; fail test-with-server on a stuck shutdown (#12752)
* s3api: reject unknown elements in PutBucketNotificationConfiguration

xml.Unmarshal skips children that match no field, so a configuration
whose only child was an unmapped element parsed as empty and returned
success while nothing was stored.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test/s3/tigris_compat: fail test-with-server when the server does not stop

A passing suite returned success even when stop-server gave up with the
server still running.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-18 15:15:38 -08:00
Chris LuandClaude Opus 5.5 f5418845ce test/s3: run tigrisdata-community s3-api-compat-tests in CI (#12741)
* test/s3: run tigrisdata-community s3-api-compat-tests in CI

Runs the suite, pinned to a commit, against weed mini through a small
provider patch, and fails on any test that is not in
expected_failures.txt or on a listed test that starts passing.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* test/s3/tigris_compat: prepare the suite again when the patch changes

The checkout was reused whenever it sat at the pinned commit, so an
edited seaweedfs-provider.patch, or a checkout whose patch failed to
apply, kept being built as it was. Record the commit and the patch's
hash once the patch applies, and prepare the checkout again when either
differs.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test/s3/tigris_compat: keep the pid file of a server already running

start-server removed weed-server.pid before it found the port taken, so
a second start lost the pid of the server holding it and stop-server
could no longer stop that one. Remove the file only once the port is
free, and have test-with-server stop only a server it started.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test/s3/tigris_compat: bound the S3 port probes

A listener that accepts but never answers held curl, and with it the
readiness loop and the cleanup after it, for good. Give each probe two
seconds.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test/s3/tigris_compat: expect a listed test to end with its listed status

A listed test passed the check with any non-pass status, so a setup error
that skipped the call under test (RestoreObject's PutObject, say) went
unnoticed. Entries now expect fail unless they name another status; only
GetBucketAccelerateConfiguration, whose setup PUT is unsupported, expects
error.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test/s3/tigris_compat: refuse to start while the recorded server still runs

A second start-server before the first server answered on the S3 port
passed the port probe and dropped the pid file of the live server, so
stop-server could no longer stop it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test/s3/tigris_compat: recognize the test server by its command line

start-server and stop-server took any process named weed at the recorded
pid for the test server, so a weed process that reused the pid would be
reported as running and killed by stop-server. Match the command line
start-server launches instead.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test/s3/tigris_compat: recognize the test server by its start time

A weed mini started from another checkout has the same command line, so
a pid it reused still matched. Record the process start time next to the
pid at launch and require both before treating the process as the test
server.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test/s3/tigris_compat: PutObjectTagging_VersionId passes now

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test/s3/tigris_compat: keep the server records until the server exits

stop-server waited a fixed two seconds after SIGTERM and then removed
weed-server.pid and weed-server.started even if weed was still shutting
down, so a later stop-server could not find it and clean could delete
data it still used. Wait up to 30 seconds for the recorded server to exit
and keep the records, failing, while it still runs.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test/s3/tigris_compat: start the server on an empty data directory

A data directory left by an earlier run on other ports made the master
freeze volume growth (topology id mismatch), and every PutObject failed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-18 15:14:52 -08:00
Chris LuandClaude Opus 5.5 53bc62c773 s3api: accept the accelerate and notification configurations the bucket already reports (#12747)
* test/s3: run tigrisdata-community s3-api-compat-tests in CI

Runs the suite, pinned to a commit, against weed mini through a small
provider patch, and fails on any test that is not in
expected_failures.txt or on a listed test that starts passing.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* test/s3/tigris_compat: prepare the suite again when the patch changes

The checkout was reused whenever it sat at the pinned commit, so an
edited seaweedfs-provider.patch, or a checkout whose patch failed to
apply, kept being built as it was. Record the commit and the patch's
hash once the patch applies, and prepare the checkout again when either
differs.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test/s3/tigris_compat: keep the pid file of a server already running

start-server removed weed-server.pid before it found the port taken, so
a second start lost the pid of the server holding it and stop-server
could no longer stop that one. Remove the file only once the port is
free, and have test-with-server stop only a server it started.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test/s3/tigris_compat: bound the S3 port probes

A listener that accepts but never answers held curl, and with it the
readiness loop and the cleanup after it, for good. Give each probe two
seconds.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3api: accept PutBucketAccelerateConfiguration with Status=Suspended

GetBucketAccelerateConfiguration always reports Suspended, but the PUT
answered 501 for every request. Suspended is now accepted, Enabled
stays NotImplemented, and the accelerate subresource resolves to
s3:GetAccelerateConfiguration / s3:PutAccelerateConfiguration.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* s3api: accept an empty PutBucketNotificationConfiguration

GetBucketNotificationConfiguration always reports an empty
configuration, but the PUT answered 501, including for the empty
configuration clients send to clear notifications. An empty one is now
accepted; a topic, queue, Lambda or EventBridge destination stays
NotImplemented.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* s3api: reject accelerate and notification bodies with the wrong root

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* test/s3/tigris_compat: expect a listed test to end with its listed status

A listed test passed the check with any non-pass status, so a setup error
that skipped the call under test (RestoreObject's PutObject, say) went
unnoticed. Entries now expect fail unless they name another status; only
GetBucketAccelerateConfiguration, whose setup PUT is unsupported, expects
error.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test/s3/tigris_compat: refuse to start while the recorded server still runs

A second start-server before the first server answered on the S3 port
passed the port probe and dropped the pid file of the live server, so
stop-server could no longer stop it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test/s3/tigris_compat: recognize the test server by its command line

start-server and stop-server took any process named weed at the recorded
pid for the test server, so a weed process that reused the pid would be
reported as running and killed by stop-server. Match the command line
start-server launches instead.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test/s3/tigris_compat: recognize the test server by its start time

A weed mini started from another checkout has the same command line, so
a pid it reused still matched. Record the process start time next to the
pid at launch and require both before treating the process as the test
server.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test/s3/tigris_compat: PutObjectTagging_VersionId passes now

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test/s3/tigris_compat: keep the server records until the server exits

stop-server waited a fixed two seconds after SIGTERM and then removed
weed-server.pid and weed-server.started even if weed was still shutting
down, so a later stop-server could not find it and clean could delete
data it still used. Wait up to 30 seconds for the recorded server to exit
and keep the records, failing, while it still runs.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test/s3/tigris_compat: start the server on an empty data directory

A data directory left by an earlier run on other ports made the master
freeze volume growth (topology id mismatch), and every PutObject failed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-18 14:55:18 -08:00
Chris LuandClaude Opus 5.5 1f5cfc7552 s3api: return x-amz-version-id from object tagging on versioned buckets (#12745)
* s3api: return x-amz-version-id from object tagging on versioned buckets

PutObjectTagging, GetObjectTagging and DeleteObjectTagging now report
the version they acted on, "null" for an object written before
versioning was enabled.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* s3api: set the tagging x-amz-version-id only on success

The header was set before the tag-conditioned policy recheck, so a
denied request carried the version id on its AccessDenied response.
Also test the "null" version and require distinct version ids.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-18 14:30:55 -08:00
Chris LuandClaude Opus 5.5 b1c315b837 s3api: report no ownership controls after DeleteBucketOwnershipControls (#12748)
* test/s3: run tigrisdata-community s3-api-compat-tests in CI

Runs the suite, pinned to a commit, against weed mini through a small
provider patch, and fails on any test that is not in
expected_failures.txt or on a listed test that starts passing.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* s3api: report no ownership controls after DeleteBucketOwnershipControls

GetBucketOwnershipControls reports BucketOwnerEnforced for a bucket that
never stored a setting, so it kept reporting it after the setting was
deleted. The delete now records that it happened, and GET answers
OwnershipControlsNotFoundError until ownership is put again. How
objects are written after the delete is unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* s3api: answer DeleteBucketOwnershipControls with 204 No Content

AWS answers a successful DeleteBucketOwnershipControls with 204 and no
body; this answered 200 with an empty OwnershipControls document.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-18 14:25:15 -08:00
aceb089bc8 lance: per-table cleanup retention and on-demand purge of deleted rows (#12739)
* lance: per-table cleanup retention and on-demand purge of deleted rows

A vector deleted from an index stayed in storage until compaction
rewrote its fragment (only above 10% deletions) and the cleanup job's
168 h retention, set once per worker, aged out every version that
referenced it.

Let a table override retain_hours and min_versions in its Lance config
(seaweedfs.cleanup.retain_hours, seaweedfs.cleanup.min_versions), and
let it request a purge (seaweedfs.purge.requested). The cleanup job
proposes a table with a pending purge regardless of the version floor
and then rewrites every fragment with deletions, retrains the indices
(a remapped IVF_FLAT index still holds the deleted vector), keeps only
the latest version, and records the request as served. A version pinned
by a tag fails the purge and leaves the request pending.

The test scans every file of a local table for a deleted vector's key,
metadata and values, and finds none after the purge.

* lance: refuse to purge a sharded vector index

A retrain builds every index name over the whole table, so the purge
turned shards of one vector index (one name per fragment set) into
names that each covered every fragment, indexing every row once per
shard. Check before compacting and fail with the request left pending,
and retrain only the vector indices.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* lance: rebuild scalar indices in a purge

Compaction only remaps scalar indices, and a retrain leaves them alone,
so a purge reported success while a bitmap index still listed the
deleted row's value, and under stable row ids a BTree index still held
the deleted key. Build each scalar index again from the remaining rows
with the parameters it was built with.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* lance: purge through the cleanup job's retention policy

The purge built its own cleanup policy, so a deployment that has the
cleanup job keep more than the floor and the window (versions an
unleased reader may still be on) would have lost them to a purge. Clean
up through retention_policy, the one the job itself uses, with the
latest version kept.

Mark the request served only once the oldest version left is one the
rebuild wrote, and record the rebuild so a purge that has to wait for
its cleanup (a tag, or that policy) retries only the cleanup instead of
compacting and retraining the indices on every pass.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* vector: lease the version every list page reads

A list page read its version without a lease unless it was a segment
that read data and metadata twice. A purge removes every version but
the latest, and only a lease tag stops it, so a listing that opened the
version before the purge's compaction could lose its files mid-read.
Pin it as get_vectors and query_vectors do.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* lance: give the cleanup job a build's execution time

A purge compacts the table and retrains its indices inside a cleanup
job, but the job declared no runtime defaults, so the admin held it to
its 90 s default and could expire a purge mid-rebuild. Declare the same
budget the indices job uses for a retrain.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-18 14:24:28 -08:00
c7d8b106d5 mount: follow a peer unlink to the surviving hard link for closed inodes (#12737)
* mount: drop a peer-unlinked name only while its inode keeps another

Move the survivor check from followSurvivingLink into InodeToPath.RemoveAlias
so the check and the removal happen under one table lock.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* mount: follow a peer unlink to the surviving hard link for closed inodes

A peer deleting one name of a hard link only moved an open handle to the
surviving name. A closed inode kept the deleted name as its primary path, so
the next open by the kernel's nodeid failed with ENOENT.

Co-authored-by: skymazer <54621405+skymazer@users.noreply.github.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* mount: share the cached-entry freshness check of peer invalidations

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* mount: keep a name a local rename reclaimed before the peer unlink arrived

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* mount: keep a name a local rename or link gave the inode after an older peer delete

The cached-entry check only covers a cached parent. InodeToPath now records,
per hard-linked inode, the newest name a local rename or link gave it with
the filer stamp, and the closed-inode peer-delete path keeps that name when
the delete is older. The record lives in a side map dropped on Forget, so
InodeEntry keeps its size class.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* filer: sign rename stream responses with the filer signature

A rename response carries the stamp of the filer that applied it, so the mount
can tell which clock the stamp belongs to, as it does for create acks.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* mount: order a peer delete against a local rename or link only on one filer clock

The stamp of a local rename or link comes from the filer that applied it, and
a delete logged by another filer runs on a different clock. Keep the name only
when both carry the same nonzero filer signature; otherwise drop it as before.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* mount: drop the local-name stamp fence for uncached parents

Ordering a delayed peer delete against a local rename by filer stamp needs a
record per alias, a shared filer clock, and the open-handle path to follow;
that is a larger change than this fix. Back to the cached-entry check, which
covers a cached parent.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: skymazer <54621405+skymazer@users.noreply.github.com>
2026-11-18 14:13:40 -08:00
a3b2fd722d mount: report opens for reading with TouchAccessTime (-accessReport) (#12721)
* mount: report opens for reading with TouchAccessTime (-accessReport)

An open answered from the metadata cache never reaches the filer, so a
filer that tracks access cannot see that a cached file is still being
read. With -accessReport, the mount sends one TouchAccessTime per open
for reading, with the open time, whether the open was answered from the
cache or not. Write-only opens are not reported.

Reports go out in the background and never hold up the open: a report
that does not fit in the queue is dropped, and drops and failures are
logged once a minute. A filer that answers Unimplemented pauses
reporting for 10 minutes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* mount: address review of the access report

- Drop reports still queued when reporting pauses; the filer that answered
  Unimplemented would answer them the same way.
- Bound each filer attempt to 2s, so a filer that accepts the connection
  but never answers leaves time for the next one, and keep using the filer
  that answered.
- Fail over only when a filer cannot be reached. Any answer, Unimplemented
  included, ends the walk.
- Report an O_CREAT open of an existing file with read access, which comes
  through Create instead of Open. A new file has nothing to read and is not
  reported.
- Stop the reporter on unmount: stop ends the workers and cancels the
  report in flight.
- Say in the flag help that it has no effect unless the filer implements
  TouchAccessTime.
- Tests no longer wait a fixed time to see that nothing else was sent; they
  look at the queue of a reporter that is not started.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* mount: fail an access report over only when the filer did not answer

A status the filer sent ends the walk even when its message contains a
word the transient classifier matches, so an Unimplemented answer still
pauses reporting and a rejected request is not resent elsewhere.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* mount: keep a filer selection made while an access report failed over

A report that reached a later filer moves the mount there only if the
selection it started from is still current, so a slower report cannot
undo another call's failover.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* mount: share an access report's time with the filers not yet tried

Each attempt is capped by an equal share of the report time left, so a
row of stalled filers cannot use up the report before a healthy one later
in the list is tried.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* mount: report the path an open for reading came through

A create that opens an existing hard link reports the name it was
given rather than the inode's first known name. Open reads the inode's
path from the inode table rather than the handle's saved path, which
other handle operations may rewrite concurrently.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* mount: check the access report stop and failover tests without races

The post-stop report is checked on a reporter without workers, so no
worker can take an accepted report before the check, and the failover
test asserts that the stalled filer was tried first.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* pb: keep the shared channel when the peer answers with its own status

gRPC never raises InvalidArgument, NotFound, AlreadyExists,
FailedPrecondition, OutOfRange or DataLoss itself, so such a status is
the peer's answer. It fell through to the string matching, and a message
naming a path like /transport closed the cached channel and cancelled
every other call on it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* mount: say which filer selection an access report keeps

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* mount: bound the in-flight call in the rejected-report channel test

The call held in flight went through touchAccessTime, whose 2 s attempt
cap could expire on a slow runner and fail the test on a healthy channel.
It now uses callFiler with the test deadline. The wait for it to reach
the filer also ends if it returns early or the deadline passes, and the
blocked handler is released on every exit path.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* mount: report each file at most once per -accessReport.interval

Every read open used to send a TouchAccessTime, adding a filer metadata
call per open. A path reported within the interval (default 1h) is now
skipped; the last report times live in a bounded LRU. 0 keeps reporting
every open.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* mount: report an open again once the file was modified since its last report

Within -accessReport.interval a read open was skipped even after a write,
so the filer kept an atime older than the mtime until the interval ran
out. relatime records that first read after a write; do the same by
comparing the entry's mtime with the last report.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* mount: keep the report time apart from the mtime it saw

Recording max(now, mtime) as the last report let a file dated a day ahead
suppress reports for a day plus the interval. Record when the report was
sent and the mtime the entry had, and treat any change of that mtime as a
modification, so neither clock can stretch or shorten the interval.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: MapleWhiteZZZ <18653045+MapleWhiteZZZ@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
2026-11-18 14:11:11 -08:00
Chris LuandClaude Opus 5.5 9f413ff2c8 shell: volume.check.disk skips replicas whose live counts match (#12743)
* shell: volume.check.disk skips replicas whose live counts match

FileCount counts every write and DeleteCount every delete or overwrite
until a vacuum drops them, so a replica vacuumed while its peer was not
keeps a different FileCount forever and was re-checked on every run
although both hold the same live needles. Only treat counts as differing
when the live counts (FileCount - DeleteCount) differ too.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* shell: volume.check.disk skips equal live counts only across compaction revisions

Equal live counts at the same compaction revision mean a missed write
and a missed delete, not a vacuum-state difference, so keep checking
those pairs.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* shell: volume.check.disk real-time recheck ignores heartbeat revisions

VolumeStatus counts are newer than the heartbeat's compaction revisions,
so a vacuum in between could make a pair at the same revision look
vacuum-skewed. Apply the old count rules there.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* shell: volume.check.disk skips on live counts only when no repair could follow

Equal live counts across compaction revisions can still hide a missed
write and a missed delete. Keep checking such pairs when -syncDeleted
could propagate the delete, or when -resurrectMissingNeedles could copy
a needle into a never-vacuumed replica.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-18 14:10:47 -08:00
e037a9c1ce s3vectors: enforce AWS metadata, key and filter limits (#12738)
* s3vectors: enforce AWS metadata, key and filter limits

PutVectors accepted vector metadata of any size and key count, bounded
only by the request body limit, and keys of any length; QueryVectors
accepted filters of any size. Apply the AWS S3 Vectors limits
(https://docs.aws.amazon.com/AmazonS3/latest/userguide/s3-vectors-limitations.html),
taking the ENHANCED index values where the modes differ, since this
implementation prefilters and takes no indexMode:

- 100 KB total metadata per vector, which also bounds a single
  non-filterable field;
- 2 KB filterable metadata per vector;
- 50 metadata keys per vector;
- 1024-character vector keys;
- at most 10 nonFilterableMetadataKeys of 1 to 63 characters at
  CreateIndex;
- 100 filter constraints per QueryVectors request.

A violation is a ValidationException before the engine is called.

* s3vectors: accept metadata whose JSON is exactly at the byte limits

metadataEntrySize counts a trailing comma for every member, so starting
from both braces overcounted by one and rejected metadata that encodes to
exactly 2 KB filterable or 100 KB total.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3vectors: rewrap the metadata limits header comment

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-18 13:49:15 -08:00
Denis 3db54ee7e5 kms/openbao: refresh the AppRole token before it expires (#12734)
* kms/openbao: refresh the AppRole token before it expires

The provider logged in with AppRole once, when it was created, and
never renewed the token. After the token's TTL every SSE-KMS upload,
and every read of an SSE-KMS object whose data key was not cached,
failed with "permission denied" (HTTP 500 to the client) until the S3
server was restarted.

With AppRole authentication the provider now logs in again in the
background once two thirds of the token's TTL have passed, and retries
a failed login every 5 s. Each login is bounded by the client's
timeout, or by 30 s when request_timeout is not positive. Close stops
the refresh. A static token, or an AppRole token without a TTL, is left
as it is.

* kms: keep the OpenBao token refresh across configuration reloads

- AddKMSProvider and InitializeGlobalKMS create a provider instance of
  their own with the new NewProvider instead of the registry's cached
  one: a reload closed the cached instance and then got it back, which
  now stopped its token refresh. The new instance is created before the
  old one is closed, so a configuration that fails leaves the current
  provider working
- openbao: time each refresh from the start of the login that issued
  the token, since its TTL runs from then rather than from the response
- tests: a reload gives a new instance and closes the old one, a failed
  reload keeps the old one; the refresh survives a reload; a slow login
  response

* kms: create a replacement provider outside the manager lock

AddKMSProvider ran the provider factory, which for OpenBao is a login
that may wait for its timeout, while holding the manager lock, so every
lookup, also of unrelated providers, waited for it. The new provider is
now created before the lock is taken; the lock covers only the swap, and
the replaced provider is closed after it. Test: a lookup while another
provider is being created.
2026-11-18 11:07:09 -08:00
Chris LuandClaude Opus 5.5 94de65b3b5 s3api: answer a malformed Cache-Control on PutObject with InvalidRequest (#12733)
PutObject rejected a Cache-Control value it could not parse with
InvalidDigest, whose message is about Content-MD5, even when the request
sent no Content-MD5. CreateMultipartUpload already answers the same parse
error with InvalidRequest; PutObject now does too.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-18 11:03:57 -08:00
ebc77400ca operation: reassign an upload after a proven-expired volume JWT (#12729)
* operation: reassign an upload after a proven-expired volume JWT

* seaweed-volume: mark an expired write JWT refused before append

Mirror the Go volume server: a multipart POST whose write JWT is signed
with the current key, names this fid and fails only on exp gets
X-Seaweed-Write-Outcome: expired-jwt-before-append on its 401, so the
uploader reassigns instead of failing. Both checks use one guard
snapshot, so a key rotation between them cannot drop the marker.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* operation: keep same-FID retry for filer-proxy expiries, snapshot the write signing key, and make the RetryForever budget test fail fast

* operation: re-mint the filer credential before a proxy same-FID expiry retry

* operation: make a proxy marked-expiry terminal in the outer retry too, not only the inner same-FID loop

---------

Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-18 11:02:52 -08:00
Chris LuandClaude Opus 5.5 ecbf7af1ca mount: compare permissions against local ids without mapping them again (#12726)
* mount: compare permissions against local ids without mapping them again

Access, open and create (without -defaultPermissions) ran FilerToLocal on
entries that maybeReadEntry and maybeLoadEntry already return in local ids.
Under a uid/gid map where a local id is also a mapped filer id, the owner
was mapped twice and the check refused the file's own owner.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* mount: cache a deferred create's placeholder in filer ids

The meta cache stores filer uid/gid and maps them to local ids on every
read, but createRegularFile cached the deferred or writeback placeholder
with the caller's local ids. The second mapping in the permission checks
used to cancel that out under a swapping id map; without it the owner of
a placeholder was refused. Cache a mapped clone and keep the handle's
entry in local ids.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* mount: test that a deferred create returns its entry in local ids

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-18 11:01:30 -08:00
b64895ed0a filer: stop following a peer that stays unreachable (#12725)
* filer: stop following a peer that stays unreachable

* filer: re-track an answering peer only while it is still followed

A signature read that succeeds just before the master removes the peer
called initPeerWatermark after the removal, clearing its mark. The loop
then exits, so the watermark it re-tracked never advances or drops and
pins aggregated readers at the settled horizon.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* filer: forget a peer's unreachable clock when the master removes it

Only a successful connection cleared the clock, so every departed address
that once failed stayed in the map for the filer's lifetime, and a re-added
peer inherited the old outage. The removal now drops it, and the clock is
guarded by peerChansLock so the removed peer's exiting loop cannot start a
new one.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* filer: report the unreachable mark only when it is set

Once peerResumeRetention forgets the mark of a peer that stays
unreachable, markPeerWatermarkRemoved no longer marks it (its watermarks
are gone), but notePeerUnreachable still saw no mark and logged
"treating it as removed" on every retry. markPeerWatermarkRemoved now
reports whether it set a new mark.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test: fix the unreachable-peer subscriber's start before the baseline write

The subscriber goroutine picked its start time itself, so a baseline write
that finished first fell before the subscription and the test failed after
20 seconds with correct delivery.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-18 11:00:15 -08:00
72ac9e96b2 mount: answer an interrupted read with EINTR, not EIO (#12723)
* mount: answer an interrupted read with EINTR, not EIO

The kernel's FUSE contract (Documentation/filesystems/fuse/fuse.rst,
"Interrupting filesystem operations") lets a filesystem either ignore an
INTERRUPT or honor it by replying EINTR to the original request. Read did
neither: the FUSE cancel channel cancels the read's context, the chunk
fetch fails with "context canceled", and every error became EIO.

Interrupts come from any signal the reader handles. Go's runtime sends
SIGURG for goroutine preemption, so Go programs reading under load hit
this on healthy data: on a 128-thread 64 KiB random-read test about 250
of 5 million reads failed with EIO, and none with GODEBUG=asyncpreemptoff=1.

Read now returns EINTR when its own context was cancelled, the way
dirReadStatus already does for directory reads and lookups do since
#11717. Go's os.File retries EINTR, so those readers see no error. A
cancelled read also skips the entry refresh retry, and neither read log
line reports an interrupt as a warning or error. Failures without an
interrupt stay EIO.

Tests: TestReadInterruptedReturnsEINTR (fails on the old code with EIO),
TestReadFailureWithoutInterruptReturnsEIO; weed/mount/... passes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* mount: require the failed-read test to reach the filer

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* mount: skip the read retry once the request is interrupted

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
2026-11-18 10:58:05 -08:00
11af0e0c12 mount: move an open file's read view to the chunks a merge wrote (#12722)
* mount: move an open file's read view to the chunks a merge wrote

When a flush finds that a file's chunks take more than twice its size,
maybeMergeChunks reads the file back and writes it again as new chunks.
The commit that publishes them makes the filer delete the chunks they
replaced. The flush put the new chunks into the handle's entry but left
its read view on the old ones, so every descriptor that kept the file
open went on reading the replaced chunks. Those read back until their
volume is vacuumed. After that a read first retries the fetch of the
missing needle for about 13 s; then refreshEntryForRead installs the
filer's entry and the read is retried. The refresh skips a handle with
writes it has not flushed, and such a read returns EIO instead.

Files that are kept open and rewritten in place, such as PostgreSQL
data files and SQLite databases, are the ones whose chunks bloat, so
they are the ones that hit this.

installRewrittenChunks puts the merge's chunks into the entry and the
read view together, under the entry lock. Chunks attached while the
merge ran were not part of it and stay. Flushes without a merge are
unchanged.

TestMergeMovesReadViewOffReplacedChunks covers the handle.
TestReadOpenHandleAfterChunkMerge runs it on a real mount, with and
without an unflushed write; for it, the fuse_integration framework
starts mini on a free master port and exposes the address.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* mount: drain uploads before a merge replaces the chunks

Uploads attach their chunks without the handle lock, first to the entry
and then to the read view, and writes that landed after doFlush drained
the pages may still be uploading when the flush takes its lock. An
upload that attached during the merge could keep a chunk older than
data the merge already holds, or put a chunk the merge replaced back
into the read view. Either outranks the merge's zero-timestamp chunk.

When the chunks are bloated enough to merge, the flush now drains the
pages again while its lock holds writes off, and takes the chunks again.
FlushAll returns only after every upload has attached to both the entry
and the read view, so nothing attaches during the merge.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* mount: leave a pre-merge upload failure to the next flush

The drain before a merge returned an upload failure through the
metadata flush. doFlush then retried it as a filer error and answered
EIO, where the page flush answers a full disk with ENOSPC. The drain is
there only to wait for the uploads: a failure stays with the pages, and
the next flush reports it as a write error, as it would without a merge.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* mount: keep chunks attached while a flush installs its compacted list

A write that lands after doFlush drained the pages can still be uploading
when flushMetadataToFiler takes its chunk snapshot. Its upload attaches the
chunk without the handle lock, and the plain flush then replaced the
entry's chunks with the list compacted from the snapshot, dropping it from
the handle and from every later commit. Keep chunks attached after the
snapshot, as the merge path already does.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* mount: fail a merge's flush on a failed drain

A failed upload found by the drain before a merge was only logged, and
the flush went on to publish a size covering bytes that no chunk stores.
It now fails the flush as a pageUploadError before anything is
published; doFlush maps it with writeErrorToFuseStatus, as it does its
own FlushData failure, and does not retry it as a filer error.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* mount: clone a flush's request entry under the entry lock

The request snapshot cloned the live entry after GetEntry had released
its read lock. Uploads attach chunks without the handle lock, so an
AppendChunks that landed during the clone raced it on the Chunks field.
Clone inside UpdateEntry, as content flush recovery does.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* mount: leave a merge to a later flush while uploads are pending

Before a merge the flush drained the pages while it held the handle and
publication locks. A writeback upload retries a transient storage error
until it succeeds, so that drain could hold both locks for as long as
storage stayed down, stalling reads, writes and stat on the handle.
The flush now checks for pending uploads and, if any, publishes the
compacted chunks and leaves the merge to a later flush. With none
pending the lock still holds writes off, so nothing attaches during the
merge, and a failed upload still fails the flush as a write error.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* mount: keep room for pending uploads when a flush merges onto a moved entry

A flush that leaves the chunk merge to a later flush publishes while
uploads are still pending, and writes that landed after the drain can be
uploading on any flush. Merging a peer's inline content or a shrink into
such a publication lets the upload attach its chunk next to the inline
bytes, where reads never reach it. Tell createContentEntry when uploads
are pending so the merge keeps room for them.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* mount: test chunk merges through the filer's chunk proxy, also under load

The tests serve the filer's chunk proxy, and like the filer the fake
deletes the chunks each commit replaces, so a read view still naming
them fails:
- TestFlushMergeThroughChunkProxy/Quiet: the flush merges, and reads
  through the open handle return the file from the merged chunk alone.
- TestFlushMergeThroughChunkProxy/UploadInFlight: an older page is still
  uploading while a newer page of the same bytes has attached. The flush
  leaves the merge, reads return the newer page, and the next flush
  merges.
- TestFlushMergeRacesWithWritesAndReads: four writers, three flushers
  and a reader on one handle, with uploads delayed at random. The
  writers pause now and then, so 14 to 18 flushes a run merge and the
  others leave the merge to a later one. Every read must succeed within
  2 s, and at the end the handle and the filer must both hold exactly
  what was written.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* mount: let a held upload go when UploadInFlight fails

The subtest's proxy holds the older page's upload until the test
releases it, and only the passing path did. A failure in between left
the handler blocked, so the proxy's cleanup waited on it and the failure
surfaced as a suite timeout. Release it from a cleanup as well, once,
and bound the waits for the upload and the drains.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
2026-11-18 10:51:31 -08:00
de5b3305a7 mount: ask the filer again when a deferred create is published mid-lookup (#12720)
* mount: ask the filer again when a deferred create is published mid-lookup

A program that writes a file under a temporary name and renames it while
the file is still open (safetensors save_file does, through Rust
tempfile's persist) looks the temporary name up before the rename. When
the parent directory's listing is not cached, that lookup goes to the
filer. If a FLUSH publishes the deferred create while the lookup is in
flight (a forked child closing its inherited descriptor sends one), the
filer can answer not-found from a read taken before the create
committed. By the time the answer arrives the flush has cleared the
handle's dirty flag, so lookupEntry finds nothing local to explain the
answer, logs "possible coherence bug" and returns ENOENT, and the rename
fails although the file exists.

Note before sending the request whether the inode had unpublished
metadata (a dirty handle or an async flush in flight). On not-found, if
it had and has since been published, ask the filer once more: the filer
acknowledged the publish before the second request is sent. A second
not-found stands, so a file another mount removed is not brought back.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* mount: judge a placeholder's not-found by one look at its base

Review of the previous commit found two problems. The retry check and the
not-found fallback read the handle separately, so a flush that finished
between the two reads still turned the answer into ENOENT. And the
pre-request read of dirtyMetadata, a plain bool that Write and the flush
set under the handle lock, raced with them on every lookup that reached
the filer.

Note instead whether the handle is a local placeholder: a deferred create
has no filer base (baseEntry) until a flush publishes it. A not-found for
a placeholder is judged by one atomic look at that base. With none, the
filer cannot have the file yet and the handle answers. With a base, the
filer acknowledged the publish, so ask once more. dirtyMetadata is not
read for this, and the not-found fallback keeps the existing check, with
the placeholder decision added.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* mount: answer a placeholder from its own handle; a second miss is final

Follow-ups from review of the previous commit:

- Pick the placeholder after the async-flush wait, from the path's
  current mapping.
- A placeholder that still has no base answers from the observed handle
  itself, instead of through the not-found fallback, which re-resolved
  the path and could miss once the handle was released and the cache
  entry evicted.
- A not-found from the second request is final. The flush installs the
  base before it clears dirtyMetadata, and keeps it set when a write
  landed meanwhile, so the fallback could return a file another mount had
  removed.
- Test gates release on cleanup, so a failing test leaves no handler
  blocked.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* mount: read a placeholder's entry the way GetAttr does

The previous commit answered a not-found for a placeholder from the
handle's live entry under entryLock. Neither of the entry's writers takes
that lock: Write and the metadata flush change size, times and chunks
under the handle lock, and async upload workers append chunks under the
LockedEntry lock, so the lookup could read an entry while it changed.

Answer with the copy the create cached when it is still this file's
(same inode), as the not-found fallback already prefers. Otherwise copy
the handle's entry under both writers' locks, in GetAttr's order (handle
lock shared, then the LockedEntry lock), and convert the copy. No caller
of lookupEntry holds a handle lock.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* mount: cache a deferred create's placeholder in filer ids

The meta cache stores filer uid/gid and maps them to local ids on every
read, but createRegularFile cached the placeholder with the caller's local
ids. Under a uid/gid map whose local id is also a mapped filer id, a lookup
or stat answered from the placeholder reported the wrong owner. Cache a
mapped clone and keep the handle's entry in local ids.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* mount: assert the placeholder is cached before checking its owner

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* mount: check the snapshot owner after evicting the cached placeholder

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
2026-11-18 10:50:28 -08:00
04e135c21e fix(helm): separate volume DNS discovery from ingress traffic (#12719)
* helm: set publishNotReadyAddresses on the volume headless service

* fix(helm): separate volume DNS discovery from ingress traffic

* test(helm): cover volume DNS discovery and ingress routing

* fix(helm): rename the volume routing Service so volumes.client cannot collide

* docs(helm): name the ready-only Service for explicit volume backendRefs

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(helm): fail when the volume routing Service name collides after truncation

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* docs(helm): point explicit volume backendRefs at the rendered client-volume name

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-18 10:49:35 -08:00
Chris LuandClaude Opus 5.5 862913d199 volume, ec_balance: clean up failed EC shard copies; stop re-proposing empty-index volumes (#12718)
* volume: remove files a failed VolumeEcShardsCopy wrote

The shard files were copied before the .ecx, so a source holding a
0-byte index failed the request only after its shards had landed, and
nothing removed them: every retried move stranded another unmounted
shard on a new disk.

Copy the .ecx first, staged beside its target, so a bad index fails
before any shard lands; rename it into place only after the shards and
sidecars have copied, so a failed copy never truncates or replaces an
.ecx the destination already has. On any error, remove exactly the
files this call created; files already on the disk (an existing
holder's shards, index or journal) are left alone. Go and Rust volume
servers behave the same.

Claude-Session: https://claude.ai/code/session_011UpvafcfUuC2mLxNAXLQJi

* ec_balance: skip EC volumes whose every holder reports an empty .ecx

A shard copy refuses a 0-byte .ecx, so detection proposed the same
failing move of such a volume every cycle. Leave volumes with no
holder reporting a needle out of planning; their shards still count
toward disk occupancy.

Claude-Session: https://claude.ai/code/session_011UpvafcfUuC2mLxNAXLQJi

* volume: serialize VolumeEcShardsCopy per volume

Overlapping copies of one vid shared the staged .ecx, and a failed copy's
cleanup removed shards that a concurrent copy had already committed and
mounted. Hold a per-vid EC lifecycle lock across the whole copy and its
cleanup, named as in the enterprise build (Store.EcLifecycleLock /
ec_lifecycle_lock).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* volume: overwrite an existing .ecx in place when committing a copied index

A mounted EcVolume reads its index through an open handle, so renaming
the staged copy over it left the reader on the replaced inode: tombstones
the source had folded into its .ecx never reached the mounted holder.
An index already on the disk is now overwritten in place, as before the
staging change; a fresh destination still gets the rename. Go and Rust.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* volume: stage replacements of shards a VolumeEcShardsCopy finds on disk

The shard transfer opened an existing destination shard with O_TRUNC and
removed it on a stream or write error, so a failed retry deleted a shard
this disk already held. A shard already present is now copied to
<shard>.copying and renamed into place only after every other file has
copied; a mounted copy is reopened so it serves the new file. Fresh
shards still write in place and are removed on failure. Go and Rust.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* volume: stage the .vif and .ecsum a VolumeEcShardsCopy brings until commit

The sidecars landed as soon as they transferred, ahead of the deferred
.ecx commit, so a failure after them left a new .vif beside the index
the failed copy kept. They are now staged as .copying and renamed into
place with the shards and index once every transfer has succeeded; a
source without one still stages nothing. Go and Rust.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* volume: run a Rust VolumeEcShardsCopy to completion when its caller goes away

tonic drops a cancelled handler future at its current await, which
released the EC lifecycle lock while file operations the copy had started
(a cleanup remove_file on the blocking pool, a create or rename) could
still land; a retry taking the lock could then lose its files to them.
The copy now runs in a spawned task that owns the lock, as Go's handler
runs to completion.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* volume: fail the sidecar copy tests with an unreadable source .ecsum

A directory at the source path reads as "not found" on volume servers
that skip non-regular EC files, which turned the failure into a no-op.
A mode-000 file fails the transfer on every implementation; root and
Windows, which can read it, skip the test.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* volume: copy each shard a VolumeEcShardsCopy lists once

A shard listed twice and already held here was staged twice; the first
commit rename consumed the staged file, so the second failed the request
after the shard was replaced, and every retry failed the same way.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* volume: wait out an in-flight VolumeEcShardsCopy before mounting its shards

A shard the copy had already landed could be mounted while the copy was
still running; the copy's failure cleanup then removed the file but left
the shard registered and announced. MountEcShards takes the per-volume
lifecycle lock, as the enterprise build already does, and the Rust mount
handler takes the same lock.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* volume: fail a VolumeEcShardsCopy whose replaced mounted shard cannot be reopened

The copy reported success while the mounted shard kept serving the
replaced file. The error lets the caller retry the copy instead of
treating the shard as delivered.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* volume: check which transfer fails in the EC copy cleanup tests

An earlier failure, before any shard or sidecar landed, would leave the
destination unchanged and pass every assertion without exercising the
cleanup. The sidecar test also skips on Windows, where a mode-000 file
stays readable.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* volume: fail the sidecar transfer in the EC copy test by injection

A mode-000 file is still readable to root and on Windows, so the test
skipped there. The source now returns an error for the .ecsum request
only, and the test checks that it served .vif before that failure.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* volume: take the EC lifecycle lock for a blanket full teardown

A teardown acknowledged while a detached copy of the same volume was
still waiting on its source let that copy recreate the index and shards
afterwards. Go and Rust now wait for the copy, as the enterprise build
does.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* volume: keep the request id on a Rust EC shard copy

The copy now runs in a spawned task, outside the gRPC layer's request id
scope, so its CopyFile calls to the source went out with a new id. Carry
the caller's id into the task.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* volume: unmount a replaced EC shard whose reopen fails

The failed copy left the mount reading the replaced file's inode, so the
destination kept serving the old shard. Unmount it, which also reports
the removal to the master, so a retry copies and mounts it again.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* volume: make the Rust EC copy tests fail for the right reason

The sidecar test now serves a directory as the source .ecsum, which root
cannot read either, so it no longer returns early there. The cancel test
holds the store until the handler's join reports the cancellation, so a
copy that finished before the abort can no longer pass it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* volume: stage every shard a VolumeEcShardsCopy transfers

A shard missing here was written in place with a cleanup guard armed
before its transfer started, so a failed copy removed a shard that
ReceiveFile wrote meanwhile. Every shard is now staged, and only one
that did not exist when it is moved into place is removed on failure.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* volume: take the EC lifecycle lock for VolumeEcShardsRebuild

A rebuild folds the deletion journal into .ecx. Run during a copy of the
same volume, the copy's index then replaced the folded tombstones. Go
and Rust take the lock the enterprise build already takes here.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* volume: check the staged .vif bytes in the EC sidecar failure test

A source without the optional .vif also returns success, so the served
extension list could not prove the transfer. The test now reads the
staged .vif when the .ecsum request fails and compares it with the
source's.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* volume: unmount a failed reopen's shard on the copy's disk only

UnmountEcShards walks every disk, so a reopen failure on the destination
also unmounted a healthy copy of the same shard on a sibling disk. The
copy now unloads the shard from its own disk and reports that disk's
removal.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* volume: take the EC lifecycle lock for an ordinary EC shard delete

Shards a copy has staged are invisible to the delete's last-shard check,
so deleting the only canonical shard removed the index the copy was
about to commit its shards against. Go and Rust deletes now wait for an
in-flight copy of the same volume.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* volume: leave a mounted shard's copied file in place on copy failure

A shard mounted on the copy's disk whose file was missing got the copied
file reopened into its mount, then that file was unlinked by a later
failure's cleanup while the mount kept serving it. Only an unmounted
shard's new file is removed now.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* volume: skip repeated shard ids with a seen set

Scanning the request prefix for each id made a request with many
repeats quadratic while the copy held the volume's lifecycle lock.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* volume: make the Go EC copy and lock tests observe their effects

The fake source counts requests per extension, so the duplicate-id test
proves .ec05 is fetched once, and returns fixture errors instead of
failing the test from its handler goroutine. The staged .vif reaches
the test through a channel. The mount and teardown lock tests seed real
files and check that nothing happened while the lock was held and that
the operation took effect after it was released.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* volume: poll the handler directly in the Rust EC lifecycle lock tests

A spawned handler that had not run when the 200 ms timer fired passed
the blocking check without ever reaching the lock. The tests now poll
the pinned handler future themselves, which guarantees it is waiting on
the lock, and cover the ordinary shard delete as well.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* volume: observe the lifecycle-lock waiter in EC copy lock tests

The mount, teardown, shard-delete and rebuild tests slept 200 ms while
holding the lock, so a handler scheduled only after the unlock passed
without ever taking it. Wait until the handler's goroutine is blocked on
the mutex instead; each test now fails within its timeout if the handler
skips the lock.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* volume: check the shard is staged before the failure-path write in a copy test

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* volume: recover a missing EC index under the volume's lifecycle lock

Missing-index recovery fetched the index and then ran a store-wide mount
of every orphan shard, without the per-volume lifecycle lock. It could
mount shards an in-flight VolumeEcShardsCopy of another volume had just
renamed in, and that copy's failure cleanup then unlinked a mounted shard.
Each recovered volume now holds its lifecycle lock across a re-check, the
fetch and a mount restricted to that volume.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* volume(rs): recover a missing EC index under the volume's lifecycle lock

Mirror of the Go change: recover_missing_ec_indexes takes the volume's
ec_lifecycle_lock, re-checks that the index is still missing, fetches it,
and mounts only that volume's shards through mount_recovered_ec_shards_only,
so an in-flight VolumeEcShardsCopy of another volume never has its
renamed-in shards mounted from under it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* volume: mount a volume whose index a copy installed during recovery

Recovery re-checks under the lifecycle lock that the index is still
missing. When a copy installed it in the meantime, the re-check skipped
the mount too, and the volume's other shards stayed unmounted because
later scans no longer list it. Skip only the fetch and still mount the
volume's orphan shards under the lock, in Go and Rust.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* volume: fetch a recovered EC index into the disk the locked re-check found

The re-check under the lifecycle lock only kept a yes/no answer, so the
fetch still wrote to the destination from the first scan, which may be a
disk that has since gone away. Fetch into the record the re-check returned.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* volume: mount a copy-installed EC index even when no peer is known

Recovery skipped the locked step when the master was unreachable, the
lookup failed or no peer was listed. A copy that installed the index after
discovery then left the volume's other shards unmounted, and later scans
no longer list it. Always run the locked re-check, and give up for lack of
peers only when the index is still missing.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* volume: grow the stack buffer before scanning for a lock waiter

A full buffer can cut off the handler goroutine's stack, so the test would
report a handler that is waiting as one that never waited.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* volume(rs): fetch a recovered EC index into the disk the locked re-check found

Mirror of the Go change: fetch into the record the re-check under the
lifecycle lock returned instead of the destination from the first scan.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* volume(rs): mount a copy-installed EC index even when no peer is known

Mirror of the Go change: a failed lookup or an empty peer list no longer
skips the locked re-check, so a volume whose index a copy installed after
discovery still gets its orphan shards mounted.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* volume(rs): run missing-index recovery to completion when the mount RPC is cancelled

Dropping the mount handler mid-recovery released the volume's lifecycle
lock while the .ecj merge it had started on a blocking thread was still
landing, so a teardown could remove the volume before that late write.
Run recovery on its own task, as the copy handler does, so the lock is
held until it finishes; Go's handler already runs it to completion.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* volume: skip other volumes before stat in a single-volume EC scan

Recovery re-checks and mounts each volume with its own scans, so a
request with many orphans repeated whole-store discovery once per volume.
The index-owner and orphan-shard scans now take the same volume filter
and drop other volumes' files by name, before the per-file stat, and the
locked re-check scans only its own volume.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* volume(rs): skip other volumes before stat in a single-volume EC scan

Mirror of the Go change: the index-owner and orphan-shard scans take the
volume filter and drop other volumes' files by name before the per-file
stat, so recovery's per-volume re-check and mount no longer repeat
whole-store discovery under the store lock for every orphan.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* volume(rs): announce shards recovered after the mount RPC was cancelled

Recovery now outlives a cancelled mount RPC, but the heartbeat nudge
stayed in the handler, so shards it mounted after the caller left waited
for the next full EC heartbeat. Nudge it from the recovery task, as Go's
loads announce each recovered shard.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* volume: match the collection as well as the volume id in the recovery re-check

Discovery keys orphans by (collection, vid), but the locked re-check
matched the vid only. Two collections with orphans of the same vid could
both resolve to the first record, so the other's index was never fetched.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* volume(rs): match the collection as well as the volume id in the recovery re-check

Mirror of the Go change.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-18 10:48:21 -08:00
Chris LuandClaude Opus 5.5 8c6d09f498 s3api: answer NoSuchBucket from GetBucketRequestPayment for a missing bucket (#12716)
* s3api: answer NoSuchBucket from GetBucketRequestPayment for a missing bucket

The handler wrote the static BucketOwner answer without resolving the
bucket, so a bucket that does not exist looked like one that does.
Check the bucket first, as PutBucketRequestPayment and the other static
configuration getters already do.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3api: test GetBucketRequestPayment on existing and missing buckets

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-17 22:38:48 -08:00
eb1dca8088 filer: release the stamps of parent directories a rename creates (#12711)
* filer: release the stamps of parent directories a rename creates

* filer: test that a suppressed create keeps the moved entry's stamp in flight

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-17 22:37:56 -08:00
deacd575ea operation: prepare upload content before assigning the volume (#12714)
* operation: prepare upload content before assigning the volume

* operation: keep the existing upload function names for prepared payloads

UploadData and doUpload prepare once and hand the prepared payload to
retriedUploadData, so the retry loop, doUploadData, uploadChunk and
uploadChunkToHolders take a preparedUpload directly instead of growing
a prepared twin of each and branching on nil.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* operation: send ciphertext whenever the prepared payload carries a cipher key

doUploadData chose the cipher branch from the option, so a payload
prepared under a different option could be sent as plaintext metadata
or lose its key. Decide from the prepared payload itself.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* operation: build a chunk's preparation and upload options in one place

The chunk loop prepared with a hand-written option while uploadChunk and
uploadChunkToHolders built their own, so a setting that shapes the bytes
could reach one and not the other. chunkUploadOption now feeds all three.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* operation: test that a reassigned chunk reuses its prepared payload

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-17 22:36:46 -08:00
Chris LuandClaude Opus 5.5 194e6ede9f seaweed-volume: reject FetchAndWriteNeedle reads that miss the requested size (#12717)
The Go volume server fails the RPC when the remote read returns a
different length than req.Size; the Rust server cached whatever came
back as a full-size chunk.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-17 22:35:38 -08:00
97f49571a1 wdclient: let master-pushed adds replace the gateway's lookup-fed locations (#12705)
* wdclient: let master-pushed adds replace the gateway's lookup-fed locations

After a master leader change the S3 gateway clears its FilerClient cache and
refills it through filer lookups. The filer's own master client may still
answer from the previous leader's generation until the new leader has heard
from the volume servers, so a volume server lost together with the old leader
is cached again. The new leader never knew that server and never pushes its
removal, and the gateway keeps trying it until reads fail on it volume by
volume, one 30 s dial timeout at a time.

The gateway's master client relearns every volume from the new leader's
adds. On each add, cache the volume where the master client now knows it to
be, in place of what a lookup cached, and keep lookups already in flight from
caching their answer over it. A provider answer for a volume is now cached
whole or not at all.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* wdclient: take a lookup's fill epoch before its second cache check

A master add that landed between the second cache check and the read of
the fill epoch was invisible to both: the check had already missed, and
the epoch the lookup then read already included the confirmation, so a
filer answer that still named the lost server was merged back into the
confirmed entry. Taking the epoch first means such an add either shows
up in the check or stops the answer from being cached. The same order
covers a reset or clear in that gap.

TestConfirmDuringLookupIsNotUndone fires a confirmation from a test hook
at that point and fails with the old order.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* wdclient: gate the master lookup provider's cache writes on the map epoch

The provider wrote each answer straight into the vid map, so a lookup sent
before a master reset and answered after it landed in the new generation,
which a gateway's FilerClient now reads as the current master's view. The
answer now goes through fillLocations, which drops it once a reset passed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* wdclient: keep the vidMapClient doc comment on its type

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
2026-11-17 19:10:20 -08:00
Konstantin Adzer 91eaabf371 s3: answer ClientDisconnected for a GET the client aborted before the first byte (#12707)
* s3: answer ClientDisconnected for a GET the client aborted before the first byte

Fixes #12706

* s3: tell a gone client on TLS and HTTP/2 too, and not on a connection the server closed
2026-11-17 17:15:00 -08:00
c91da3d1f8 filer: fail health check when absent from lock ring (#12709)
* filer: fail health check when absent from lock ring

If the master broadcasts a lock ring that does not include this filer
(e.g. the late-cleanup race in seaweedfs#11442), every lock acquisition
fails with "no lock server found" and locked writes, including S3 PUTs,
hang indefinitely while /healthz and /readyz still report OK.

Make the filer health handler return 503 once this filer has been
missing from its own lock ring for longer than a 30s grace period, so
Kubernetes liveness probes restart it and it re-registers with the
master. The grace period covers startup before the first ring update
and the master's ring-change batching delay.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* filer: hold the lock-ring health clock until the filer announces

AnnounceOnceCaughtUp can hold the master registration for up to a minute
while HTTP already serves, so the filer cannot be in the ring yet. Report
healthy until the announce gate opens and start the grace period after it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* filer: keep a probe racing a ring rejoin from reading a reset clock

A probe that loses the CompareAndSwap to a concurrent healthy probe could
load the reset zero and report the filer absent since the epoch.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* filer: trim the lock-ring health comments

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* filer: restart the lock-ring health grace when a ring update brings the filer back

The clock only reset when a probe saw the filer in the ring, so a rejoin
and a second removal between two probes reused the first absence and
failed the next probe early. Reset it from the ring snapshot callback.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* filer: test the announce gate leaves the lock-ring health clock unset

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* filer: re-read lock-ring membership after starting the health clock

A probe whose first read raced a rejoin could start the clock after the
snapshot callback reset it, leaving a stale start for the next removal.
The ring is installed before the callback runs, so a second read sees the
rejoin whenever the reset came first.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* filer: treat a health clock changed during the probe as a rejoin

A rejoin and a second removal can both land between the probe's two
membership reads. The second read then shows the filer absent again, and
the probe judged the new absence by the expired start it had read before
the rejoin's reset. The snapshot callback resets the clock before the next
ring update is installed, so a clock that no longer matches the probe's
start means a rejoin happened; report healthy and let the next probe use
the fresh clock.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* filer: test ring updates that land during a lock-ring health probe

Move the absence clock into lockRingAbsence, which takes the membership
read as a function, so the test can install ring updates between the
probe's reads: a rejoin after the first read, and a rejoin plus removal
between the two reads.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* filer: give every lock-ring health reset a new clock value

A probe that saw the filer absent started the clock with a CAS from 0,
so a rejoin that reset the clock to 0 again before that CAS was
invisible, and the probe could install a start time from before the
rejoin. Each reset now stores a new non-positive value and the probe
swaps from the value it loaded before the membership read, so any
rejoin during the probe makes the CAS fail. This drops the second
membership read and never waits on the LockRing mutex.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* filer: test lock-ring rejoins before and after the probe's membership read

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* filer: keep the lock-ring version lookup out of the unhealthy health path

LockRing.Version takes the ring mutex, which SetSnapshot holds while
the snapshot callback transfers locks, so an expired-absence probe
could wait on those transfers and time out instead of answering 503.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* filer: test an expired-absence health probe while the ring mutex is held

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
2026-11-17 17:13:45 -08:00
c76fb36416 test(fuse): two mounts staging the same files without -dlm (#12704)
* test(fuse): two mounts staging the same files without -dlm

Two jobs that extract the same archive into one shared directory, each
through its own mount and without -dlm, both create many of the same
files: a file that exists is skipped, otherwise it is created, written
and closed, then chmod'ed and given its mtime. Before #12593, #12594 and
#12596 the attribute update and the content flush sent the mount's whole
cached entry. Sent after the other mount had rewritten the file, the
update put back chunks that the rewrite had dropped and the filer had
deleted, so the entry pointed at deleted needles and the file failed to
read once its volume was vacuumed. Nothing told the writers.

TestStageSameFilesWithoutDLM runs that on the fuse_dlm cluster with the
mounts started without -dlm (startTestCluster takes a dlm flag;
startDLMTestCluster is unchanged) and checks every staged file: each
chunk is still on the volume server (HEAD on the fid, without
readDeleted) and the filer serves the bytes written. What the losing
writer is told depends on the version and is only logged.

With a 4.53 weed binary the test fails: 66, 71 and 70 of the 400 files
were broken in three runs. With master it passes in about 48 s.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test(fuse): bound the staging test's filer and volume checks

The checks used the default HTTP client and a background context, so a
server that accepts a request and stops answering held the test until the
package timeout instead of failing the check.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test(fuse): fail the staging test unless both mounts wrote the same files

A mount whose MkdirAll failed returned early, and a mount that started
late could skip every file, so the test passed with one writer. Start both
loops together, fail on setup errors, and require files both mounts wrote.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test(fuse): bound the staging test's checks with one deadline

A per-call timeout still let 400 hung lookups and reads run for hours.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
2026-11-17 17:08:39 -08:00
0d3f5a220c s3: keep serving for -preStopSeconds after SIGTERM (#12703)
* s3: keep serving for -preStopSeconds after SIGTERM

On SIGTERM the gateway closed its listeners at once, so requests a load
balancer or Kubernetes Service still routed to it failed until the endpoint
was removed. With -preStopSeconds set, the gateway makes /status, /healthz
and /readyz return 503, keeps serving for that many seconds, then runs the
existing graceful shutdown. 0 (the default) keeps the old behavior.

The flag is registered for weed s3 and as -s3.preStopSeconds for weed
filer, weed server and weed mini. A cancelled weed mini cluster context
ends the wait early. The Helm chart passes s3.preStopSeconds and
filer.s3.preStopSeconds through.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3: assert the preStopSeconds chart render under set -e, drop timing from the probe test

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3: wait before weed mini cancels its clients; size grace periods in the chart

weed mini's interrupt hook runs before the S3 one and cancels the S3
gateway's shutdownCtx, so the pre-stop wait never happened there. The wait
is now also registered with beforeMiniClientsShutdown, which runs before
that cancel, and a sync.OnceFunc keeps the interrupt hook from waiting a
second time. Outside weed mini the registration is a no-op.

Helm: the s3 deployment fails to render when terminationGracePeriodSeconds
is below preStopSeconds + 15. The filer statefulset and the all-in-one
deployment, whose grace period is fixed at 60s, add preStopSeconds to it.
allInOne.s3.preStopSeconds is passed through, inheriting s3.preStopSeconds.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3: keep 64-bit counters aligned on 386; let allInOne.s3.preStopSeconds=0 disable the wait

draining sat ahead of inFlightDataSize and inFlightUploads in S3ApiServer
and shifted them off 8-byte alignment on 32-bit builds, so the atomic
upload-limit counters panicked there. It now follows them.

Helm: allInOne.s3.preStopSeconds inherits s3.preStopSeconds only when null;
default treated an explicit 0 as unset. The s3 values and the grace period
error now say a lifecycle preStop hook's time comes on top.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* grace: add OnInterruptFirst for hooks that must run before any shutdown

Interrupt hooks run in reverse registration order, so a hook cannot run
ahead of a service that registers later, such as a volume server that is
still loading its volumes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3: run the -preStopSeconds wait before every other interrupt hook

Under weed server the volume server registers its hook once its volumes
are loaded, often after the S3 gateway. Its hook then ran first and shut
the volume server down, so the gateway spent the whole wait serving
requests that could not read or write objects.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
2026-11-17 17:06:06 -08:00
Chris LuandClaude Opus 5.5 c4e1802e3e s3: route ';' queries as authorized, and report an empty s3:prefix to policies (#12702)
* s3: escape raw semicolons before the router matches a route

registerRouter applied EscapeSemicolonsInQuery with router.Use, but mux
runs router middleware only after it has picked the route. Queries()
matchers split pairs on ';', so GET /bucket?uploads;x=1 reached the
multipart listing handler while the handler and the IAM resolver read
uploads%3Bx=1 and saw no uploads subresource. Wrap the S3 router with the
escape so routing and authorization read the same query.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3api: report a listing without prefix as s3:prefix "" in policy conditions

The advanced IAM path already sets s3:prefix to "" when a bucket listing
names no prefix, as AWS does, but bucket policies and attached policies
read conditions from ExtractConditionValuesFromRequest, which leaves the
key out. The AWS home-directory policy that grants s3:prefix ["", "home/"]
then denied the root listing, and a Deny keyed on the empty prefix never
applied.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3: test the served handler composition for the semicolon escape

Move the escape and the Vectors split into s3Handler so a command test can
send ';' queries through the handler the server actually serves.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3: keep the isVectorsRequest doc comment on its function

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-17 17:05:34 -08:00
1d30b0de21 fix(s3api): complete bucket-listing IAM condition handling (#12699)
* fix(s3api): complete bucket-listing IAM condition handling

Populate s3:delimiter and s3:max-keys for bucket. Add regression tests for listing conditions and variants.

Assisted-by: gpt-6.1-sol, deepseek-v4.1-flash

* fix(s3api): ignore max-keys for multipart listing policies

Assisted-by: GPT-6.1 Terra

* s3api: drop max-keys from multipart-upload listing policy conditions

Bucket policies and embedded IAM policies read condition values through
ExtractConditionValuesFromRequest, which still exposed max-keys on ?uploads
listings. Those page by max-uploads, so an unused max-keys parameter could
satisfy an s3:max-keys condition. Match the advanced IAM path and leave the
key absent there.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-17 17:05:00 -08:00
5681be241e topology: never report negative free volume slots (#12698)
* topology: never report negative free volume slots

Free volume slots are computed as max + remote - volume - ecSlots. When phantom or over-registered volumes push the count past the configured max, the result is negative and rolls up into the cluster total, causing volume grow and assign to fail with a negative 'volumes left' count even while healthy servers have capacity. Clamp every free-volume count at zero so an over-committed server contributes no slots instead of subtracting from the cluster.

* topology: drop the issue reference from the free-slot test comment

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* topology: test free slots next to an over-committed server

A server with more volumes than slots must not hide the free slots of
the other servers in its rack, data center or topology.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* topology: sum free slots per data node in racks, data centers and topology

Clamping the rolled-up counters only clamped their sum, so one server with
more volumes than slots still hid the free slots of every other server in
its rack, data center and topology. Interior nodes now add up their
children's free slots, so each data node counts as zero at worst. A data
node in maintenance mode no longer adds slots to its parents either.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* topology: test that disk info keeps an over-committed deficit

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* topology: keep the signed free volume count in disk info

EC encode retries and the balance planners add refunds and planned moves
to the published FreeVolumeCount. Clamping it at zero dropped an
over-committed disk's deficit, so a refund could make a full disk look
free. Volume growth already reads the clamped AvailableSpaceFor.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* topology: test placement next to an over-committed server

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* topology: accept either data center order in the placement spread check

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-17 17:02:01 -08:00
8ba33fae71 rust volume: bulk-load needle maps on redb 4.6 and compact the old floor (#12695)
* seaweed-volume: expect a fresh redb file to start at 72 KiB

The assertion fails on redb 4.2.0, whose empty database is about 1 MiB.

* seaweed-volume: bulk-load redb rebuilds with the stable cursor

redb 4.6.0 has no experimental_cursor feature. Full rebuilds insert
sorted needle ids through insert_before, and a new .rdb starts at 72 KiB.

* seaweed-volume: test which redb files are still at the old floor

The gate is missing, so the test does not compile yet.

* seaweed-volume: decide when a reused redb is still at the old floor

Compact only a file above 72 KiB and at most 1 MiB plus 8 KiB, and only
when floor_compact has not been recorded.

* seaweed-volume: test one-shot compact of an old-floor redb

Reuse, shrink, and storage-error recovery are specified. The helpers
they call are not implemented, so the tests do not compile yet.

* seaweed-volume: compact a reused redb that is still at the old floor

One compact runs when the file is above 72 KiB and at most 1 MiB plus
8 KiB. A storage error drops the handle and opens the same path again
so the volume does not keep a PreviousIo latch.

* seaweed-volume: publish a redb replay before compacting the old floor

compact() is what would make a Durability::None tail durable, and a
storage error then drops that tail on reopen. An in-band file is
committed first. The floor_compact flag is read only inside the band,
so a meta error cannot rebuild a larger index.

* seaweed-volume: drop the redb experimental cursor CI step

redb 4.6 has no redb-experimental-cursor feature. The needle-map tests
that used it now run in the normal cargo test step.

* seaweed-volume: drop the experimental-api-5 notes on redb reads

redb's experimental-api-5 can no longer be enabled, so these reads always
use the inherent ReadOnlyTable::get.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* seaweed-volume: drop the always-false flag from the old-floor band check

Both callers pass false and read floor_compact on their own, so the band
check only needs the file length.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-17 17:00:55 -08:00
d8a47042c5 master: unregister a volume server whose heartbeats stop (#12697)
* master: unregister a volume server whose heartbeats stop

A volume server sends a heartbeat every 5 s. When its host hangs or its
network drops packets, its SendHeartbeat stream stays open, and the
master unregisters it only once gRPC keepalive closes the connection,
60 s plus 20 s after the last frame. Until then the master keeps
assigning writes to volumes with a copy on it, and those writes hang.

SendHeartbeat now ends the stream when no heartbeat arrives for three
pulses (15 s), which unregisters the server through the existing
deferred path and tells clients its locations are gone. The server
reconnects and registers again once it can reach the master.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* master: test that clients learn a silent server's volumes are gone

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* master: fail the silent-stream receive test instead of hanging

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
2026-11-17 16:59:40 -08:00
Chris LuandClaude Opus 5.5 dbb2fdb93b pb: recover when a local gRPC socket file is deleted (#12694)
* pb: fall back to TCP when the local gRPC socket cannot be dialed

A temp cleaner can delete a local gRPC socket file while its server keeps
running. Every later dial to that server then failed with "no such file or
directory", so the master could no longer reach its own volume server and
volume growth stopped. Dial the server's TCP address when the socket dial
fails, as the S3 gateway already does for the filer's HTTP socket.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* pb: recreate a local gRPC socket whose file was removed

Once its socket file was deleted, a server kept accepting on an
unreachable listener until restart, so same-host RPCs stayed on the TCP
fallback. Check the path every minute and, when the file is gone, close
the old listener and bind the socket again.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* pb: retry a failed local socket rebind on the next check

A failed rebind ended the watcher, so the socket stayed missing after the
cause cleared. Bind the replacement before closing the old listener: a
failed bind keeps the old listener serving and is retried on the next
check, and the old Serve still reports server shutdown. The old listener
no longer unlinks the path on close, since the replacement now owns it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* pb: set the local socket unlink policy before serving the listener

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-17 16:58:19 -08:00
b74971b6d4 server: add -grpc.localSocketDir for the local gRPC sockets (#12693)
* server: add -grpc.localSocketDir for the local gRPC sockets

`weed server` and `weed mini` hard-coded each local gRPC Unix socket
under /tmp. systemd-tmpfiles ages /tmp after 10 days and macOS after 3,
so a long-running server loses its socket, and RegisterLocalGrpcSocket
keeps routing local dials to the missing path: the master can no longer
reach its volume server.

Add -grpc.localSocketDir to both commands, defaulting to /tmp so an
existing setup is unchanged, and build every socket path from it with
filepath.Join. A folder that does not exist or is not a directory is
refused at startup rather than at the first local RPC. A folder whose
socket paths exceed the platform's sockaddr_un limit is refused too,
because every bind then fails with EINVAL while the server keeps running
and each local RPC dials a socket that does not exist.

Refs #12692

* server: refuse a -grpc.localSocketDir the server cannot bind in

An existing folder the server's user cannot write to passed the startup
check, and then every socket bind failed: ListenOnLocalSocket only logs
the error while GrpcDial keeps routing local dials to a socket that was
never created. After the path-length check, create and remove a probe
Unix socket in the folder, and refuse startup with the bind error if
that fails.

* server: give the -grpc.localSocketDir probe a random name

A probe named only by the process id could be precreated by another
user in a shared /tmp, or left by a crashed process whose pid was
reused, and the server cannot remove such a file, so startup failed
although the real socket paths were free. Name the probe with 8 random
bytes, retry a name that is taken, and rely on the listener's close to
unlink it.

* server: keep the -grpc.localSocketDir probe name within the service socket lengths

The probe was 5 bytes longer than the master and volume socket names, so a
folder whose service sockets fit the sockaddr_un limit could still be refused.
A 23-byte probe name is shorter than any service socket name, and a process
with no local sockets skips the check.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* server: trim the -grpc.localSocketDir comments and drop issue references

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Brent Pease <179639673+brentDSL@users.noreply.github.com>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-17 16:56:15 -08:00
Chris LuandClaude Opus 5.5 95f46c8fff admin: manage filer webhook targets from the UI with live reload (#12690)
* webhook: share the default target settings

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* webhook: managed webhook targets with a drainable queue

Targets kept as JSON (name, endpoint, token, filters, retry and worker
settings) build their own queue; Close waits for queued and in-flight
deliveries before stopping it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* filer: run webhook targets stored in /etc/seaweedfs when notification.webhook.managed is set

The targets in /etc/seaweedfs/notification_webhooks.json start with the
filer and are rebuilt from the store whenever the file changes on any
filer, alongside notification.toml. The previous queue drains before it
stops. Events for the file itself, which holds bearer tokens, are not
sent to notification queues.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* admin: manage filer webhook targets

A Webhooks page under Management lists, adds, edits and deletes the
targets kept in /etc/seaweedfs/notification_webhooks.json. Bearer
tokens are write-only: the API reports whether one is set, and an empty
token on edit keeps the stored one.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* filer: keep the webhook queue when the stored targets did not change

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* filer: keep a replaced webhook queue open until its senders hand over

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* admin: read chunk-backed webhook targets before editing them

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* shell: keep the stored webhook targets out of fs.meta.notify

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* webhook: reject negative managed target settings

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* webhook: no signal handler on queues that reloads replace

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* filer: swap the webhook queue without waiting behind blocked senders

The read lock held across SendMessage made a reload wait for any sender
blocked on a saturated target, and the waiting writer stalled every
other filer write with it, which is when an operator disables that
target. Close now refuses new sends and waits for the senders already
inside SendMessage, and a send that hits the closing queue retries on
its replacement.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* webhook: release blocked test handlers before the server shuts down

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* filer: always run the webhook targets managed in the admin UI

Drop the notification.webhook.managed switch: every filer loads and live
reloads the stored targets. Without stored targets no queue is created.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* scaffold: place the admin-managed webhook note above enabled

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* scaffold: separate the admin-managed webhook note

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* scaffold: note that admin-managed webhooks follow the notification queue

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* filer: retry loading webhook targets after a failed read

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* filer: retire the admin-managed webhook queue on shutdown

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-17 16:54:54 -08:00
github-actions[bot] 386385101d docs: regenerate star history chart 2026-11-18 00:33:52 +00:00
Chris LuandClaude Opus 5.5 6342876392 ci: run the Python 3.9 jobs on 3.10 (#12691)
* ci: run the Python 3.9 jobs on 3.10

ubuntu-latest now resolves to Ubuntu 26.04, which has no Python 3.9 build
for actions/setup-python, so the Ceph S3 tests and the PyArrow Parquet
3.9 job fail during setup. 3.10 is the oldest version built for 26.04.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test: list the Python versions the parquet CI runs

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-16 22:07:34 -08:00
Chris LuandClaude Opus 5.5 40182a68bc admin: keep the dark theme from flashing light on reload (#12689)
* admin: apply the saved theme inline at the top of the page head

The theme script was an external file, so a reload had to fetch or
revalidate it before the root got its theme, and the page could paint
with the light default in the meantime.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* admin: set color-scheme on the root together with the theme

Until the stylesheets load, the browser paints the page canvas from the
root's color-scheme, which only the stylesheets set, so a dark theme
showed a white canvas for the duration of every reload.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-16 20:31:00 -08:00
github-actions[bot] eed00c3737 docs: regenerate star history chart 2026-11-17 01:00:14 +00:00
Chris LuandClaude Opus 5.5 b449301a4e ec: mount shards through the encode's .vif when a source volume's .vif shares the disk (#12676)
* ec: mount shards against the index .vif when the local one has no EC config

A worker encode can send shards to a disk that still holds the source
volume's .vif while the encode's .vif lands with the index on a sibling
disk. The shard disk mounted through the source's config-free .vif, so
it reported encode time 0 and fell back to the sidecar layout, and reads
carrying the encode time were rejected as a different encode run.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* volume(rs): mount shards against the index .vif when the local one has no EC config

Mirrors the Go mount: a data-dir .vif that records no EC config yields
to the idx dir's when that one records it, and load_vif_info_across_dirs
reports the directory the .vif actually came from.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* storage: do not mirror EC sidecars onto a disk whose .vif has no EC config

Startup mirroring copied .ecx/.ecj next to a source volume's leftover
config-free .vif and kept that .vif, so after a restart the shard disk
mounted self-contained with encode time 0. Leave such a disk on the
cross-disk mount, which reads the owner's .vif.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* volume(rs): do not mirror EC sidecars onto a disk whose .vif has no EC config

Mirrors the Go startup mirror: a shard disk holding a source volume's
config-free .vif stays on the cross-disk mount, which reads the owner's
.vif, instead of mounting self-contained with encode time 0.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* ec: take the encode .vif from a sibling data dir when the index dir is shared

With a separate -dir.idx the .ecx is shared by every disk, but the encode's
.vif lands in the data dir of the disk that received it. A disk still holding
a source volume's config-free .vif mounted through it at generation 0.
NewEcVolume now accepts the server's data dirs and takes a sibling's EC .vif
when the index is not local and every sibling EC .vif names the same run.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* storage: give EC mounts the data dir of every disk on the server

NewStore records every data dir on each DiskLocation before loading, so the
startup scan, cross-disk reconcile and MountEcShards all pass them to
NewEcVolume.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* volume(rs): take the encode .vif from a sibling data dir when the index dir is shared

Mirrors the Go mount: EcVolume::new_with_vif_dirs takes a sibling's EC .vif
when this disk's own records no EC config, the index is not local, and every
sibling EC .vif names the same run.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* volume(rs): give EC mounts the data dir of every disk on the server

main sets Store.store_dirs before the first add_location, so a disk loaded
before its siblings still sees them; each DiskLocation passes them to
EcVolume::new_with_vif_dirs.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* storage: size startup EC validation by the sibling encode .vif

With a shared -dir.idx, a disk that still holds the source .dat and its
config-free .vif read the default data-shard count, judged custom-ratio
shards an interrupted encode, and removed them along with the shared .ecx
before the mount could take the sibling's .vif. The startup lookup now
falls back to the same sibling .vif the mount uses.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* volume(rs): size startup EC validation by the sibling encode .vif

Mirrors the Go change: validate_ec_volume falls back to the sibling data
dir's encode .vif before judging shards beside a source .dat.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* storage: keep EC shards when sibling .vif files disagree on the encode

SiblingEcConfigVif reports when a sibling .vif cannot be read or the
siblings name different encode runs. The startup data-shard lookup then
returns 0 instead of the default ratio, and validateEcVolume keeps the
files rather than sizing custom-ratio shards by a ratio they do not use.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* volume(rs): keep EC shards when sibling .vif files disagree on the encode

Mirrors the Go change: sibling_ec_config_vif returns Err for an unreadable
sibling .vif or disagreeing runs, ec_data_shards_from_vif returns 0 then,
and validate_ec_volume keeps the files.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* storage: release the opened EC shard when its volume cannot be built

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-16 16:38:17 -08:00
Chris LuandClaude Opus 5.5 9e5357782e volume: keep reads over the download limit local when readMode is local (#12679)
* volume: keep reads over the download limit local when readMode is local

With -readMode=local the server never sends a read for a volume it does
not hold elsewhere, but a read over -concurrentDownloadLimitMB on a
replicated volume was still redirected to another replica. Skip the
replica fallback in local mode, so the read waits for a download slot
and answers 429 when the wait times out.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* volume(rs): keep reads over the download limit local when readMode is local

Mirror the Go volume server: with readMode local, a read over the
download limit waits for a slot instead of being redirected to a
replica. proxy_or_redirect_to_target no longer needs the local-mode
redirect switch.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* volume: check download capacity under the wait lock

waitForDownloadSlot read the in-flight size outside the cond lock and
the read handler broadcast without it, so a download finishing between
the check and the wait was missed and the read waited out the timeout
for a 429. Check capacity with util.WaitUntil under the lock and
broadcast under it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* volume(rs): register for the download notification before checking capacity

The wait loop created its Notified future only after the capacity
check, so a body dropped in between had its notify_waiters() lost and
the read waited out the deadline for a 429. Create the future first,
then check, as Go now checks under the cond lock.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-16 16:20:15 -08:00
f4e638fa1d Fix/admin cluster submenu (#12677)
* admin: fix cluster submenu toggle

* admin: update embedded JavaScript

* admin: drop the object store submenu handlers

The sidebar no longer renders an object store submenu, so its expand,
link and toggle handlers match nothing; the toggle handler repeated the
manual collapse that fought Bootstrap on the cluster submenu.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: ivan <i.stepanov@navigine.com>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-16 16:13:18 -08:00
Chris LuandClaude Opus 5.5 1e31c6dbf5 test(pjdfstest): check the sudo'd mount with ps, not kill -0 (#12673)
sudo switches its real uid to root while it starts, so kill -0 from the
invoking user fails with EPERM even though the mount process is alive,
and the wait loop reported "weed mount exited early" a few milliseconds
after starting it.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-16 16:12:30 -08:00
0f0f3811bf filer: serve caught-up aggregated subscribers from memory, not a held disk pass (#12674)
* filer: serve caught-up aggregated subscribers from memory, not a held disk pass

An aggregated SubscribeMetadata stream that does not take metadata chunks
(filer.sync, filer.backup, filer.meta.backup, filer.meta.tail, the
filer.remote.* syncs, the S3 gateway's subscriptions, and mounts that
predate chunk support) re-ran the persisted-log pass whenever its cursor
moved: after every memory hold at the delivery watermark and on every
2 s re-probe. The pass holds at the peers' flush low-watermark. Peers
flush on their own one-minute schedules, so on a cluster that keeps
writing some peer has always flushed past a caught-up cursor while the
slowest has not: the pass held at its first entry, the loop came back
to it, and memory was never read. The stream got each change only once
the slowest peer's flush covered it, in bursts up to a flush interval
late, and every pass listed and re-read the newest log files.

When the aggregated ring was already filling at the cursor and evicted
nothing after it, memory holds everything the persisted logs could add,
since every peer's log reaches the ring through the aggregator. Skip the
pass and read memory, held at the delivery watermark as before. The
skipped pass proves nothing past the cursor, and the next-day scan for
an empty pass does not apply. An empty ring, a ring that starts past the
cursor, and a cursor memory just refused still take the pass. Chunk
subscribers already skip it for a caught-up cursor: their ref listing
stops below the flush hold.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* filer: keep a new peer's pre-start history on the disk pass

The ring only receives a peer's events from the start of its subscription.
A peer followed for the first time after this filer started (a filer added
to the cluster, a restarted pod with a new address, or the late-arriving
peers bootstrap) starts at its notice time, so what it logged before then
is only in the persisted logs. A cursor below that start passed the ring
check and skipped the disk pass, losing those events for subscribers that
do not take metadata chunks.

Record the newest start of a peer subscription that had no resume point and
send cursors below it through the disk pass as before. Peers that rejoin with
a resume point (a restart under the same address) do not move it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
2026-11-16 16:12:06 -08:00
dependabot[bot] 53b234558e build(deps): bump actions/setup-go from 6 to 7 (#12685)
Bumps [actions/setup-go](https://github.com/actions/setup-go) from 6 to 7.
- [Release notes](https://github.com/actions/setup-go/releases)
- [Commits](https://github.com/actions/setup-go/compare/v6...v7)

---
updated-dependencies:
- dependency-name: actions/setup-go
  dependency-version: '7'
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-11-16 15:39:24 -08:00
dependabot[bot] d9c2d99f8f build(deps): bump gocloud.dev/pubsub/natspubsub from 0.46.0 to 0.49.0 (#12680)
Bumps gocloud.dev/pubsub/natspubsub from 0.46.0 to 0.49.0.

---
updated-dependencies:
- dependency-name: gocloud.dev/pubsub/natspubsub
  dependency-version: 0.49.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-11-16 15:38:44 -08:00
dependabot[bot] e3769592a5 build(deps): bump github.com/Azure/azure-sdk-for-go/sdk/storage/azblob from 1.8.2 to 1.8.3 (#12684)
build(deps): bump github.com/Azure/azure-sdk-for-go/sdk/storage/azblob

Bumps [github.com/Azure/azure-sdk-for-go/sdk/storage/azblob](https://github.com/Azure/azure-sdk-for-go) from 1.8.2 to 1.8.3.
- [Release notes](https://github.com/Azure/azure-sdk-for-go/releases)
- [Commits](https://github.com/Azure/azure-sdk-for-go/compare/sdk/azidentity/v1.8.2...sdk/storage/azblob/v1.8.3)

---
updated-dependencies:
- dependency-name: github.com/Azure/azure-sdk-for-go/sdk/storage/azblob
  dependency-version: 1.8.3
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-11-16 14:58:43 -08:00
dependabot[bot] c3f4b3e0eb build(deps): bump go.etcd.io/etcd/client/v3 from 3.7.1 to 3.7.3 (#12682)
Bumps [go.etcd.io/etcd/client/v3](https://github.com/etcd-io/etcd) from 3.7.1 to 3.7.3.
- [Release notes](https://github.com/etcd-io/etcd/releases)
- [Commits](https://github.com/etcd-io/etcd/compare/v3.7.1...v3.7.3)

---
updated-dependencies:
- dependency-name: go.etcd.io/etcd/client/v3
  dependency-version: 3.7.3
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-11-16 14:57:32 -08:00