Commit Graph
16334 Commits
Author SHA1 Message Date
github-actions[bot] 5e003bae7d docs: regenerate star history chart 2026-11-20 01:22:33 +00:00
Chris LuandClaude Opus 5.5 4ac3633f1c volume: check the needle id in the streamed-read probe and on blob writes (#12794)
* storage: reject a streamed read whose record id or size differs

The header probe before a streamed read returned not-found only when both
the record's id and its size differed from the index entry, so a record of
another needle with the same size passed and its data was streamed under
the requested key until the final checksum check.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* storage: refuse a needle blob whose header names another needle

WriteNeedleBlob compared only the blob header's size with the size it was
asked to index. A blob read from an offset that has since moved to another
record of the same size was appended and indexed under the wrong key.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* volume(rs): refuse a needle blob whose header names another needle

Mirror of the Go check: write_needle_blob_and_index compared only the blob
header's size, so a blob of another needle with the same size was appended
and indexed under the requested key.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* volume(rs): report not-found when a streamed read's record is another needle

The meta read ahead of a streamed reply took the id from the record header
without comparing it with the requested key. Return not-found when the
index entry resolves to another needle, as the Go header probe does.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* storage: check the requested id after a meta-only read

ReadNeedleMeta overwrites n.Id with the record's header id, so a meta-only
read through an index entry that resolves to another needle returned that
needle's metadata, and the streamed-read probe that follows compared the
record with itself. Return not-found when the header names another needle,
as the Rust meta read does.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test: copy a needle blob to another volume, not under another key

WriteNeedleBlob now refuses a blob whose header names another needle,
so the round trip writes the blob under its own id into a second volume
and checks that storing it under another key is refused.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* storage: compare the streamed header with a deleted entry's record size

A deleted index entry stores the negated size, so a read-deleted stream
compared the positive header size with a negative one and missed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-19 15:27:15 -08:00
Chris LuandClaude Opus 5.5 4a5dbe4c6f storage: copy an empty needle written during compaction in makeupDiff (#12796)
* storage: copy an empty needle written during compaction in makeupDiff

A size-0 index row is a live empty needle, not a delete. makeupDiff
grouped it with tombstones and appended a made-up delete marker, so
the compacted .dat lost the record's cookie and held a delete.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* volume(rs): copy an empty needle written during compaction in makeup_diff

A size-0 index row is a live empty needle. makeup_diff wrote a made-up
delete marker and a tombstone row for it, so the needle read as not
found after the commit. Copy the record like any other live row.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* storage: fail makeupDiff when the replayed needle write fails

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* volume(rs): check the compacted empty needle's id and cookie in the test

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-19 15:26:20 -08:00
Chris LuandClaude Opus 5.5 ba1bfa816c admin: show a pruned job's detail from its run record (#12795)
* admin: show a pruned job's detail from its run record

Tracking keeps the newest 1000 jobs across all job types and deletes
the details of older finished ones, while each job type's run history
keeps its last 10 successful and 10 failed runs. A job listed in the
run history could therefore open as "job detail not found". Fall back
to the run record when the job is no longer tracked.

Claude-Session: https://claude.ai/code/session_011UpvafcfUuC2mLxNAXLQJi

* admin: show 100% progress for a pruned successful run

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* admin: use the newest run record for a job across both outcomes

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* admin: keep the run record a pruned job's detail was built from

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* admin: keep looking for a run record past an unreadable history

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* admin: pick a pruned job's newest run record across job types

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-19 15:23:46 -08:00
Ping QiuandClaude Opus 5.5 be8ebc2e04 storage: build the temporary needle map in CompactByVolumeData (#12788)
* storage: red test for CommitCompact after CompactByVolumeData with a concurrent write or delete

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* storage: build the temporary needle map in CompactByVolumeData

Fixes #12783

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* storage: require a successful commit in the CompactByVolumeData commit test

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-19 15:06:05 -08:00
Chris LuandClaude Opus 5.5 bfbeed61a7 storage: read only the header and trailer in the append-time search (#12791)
BinarySearchByAppendAtNs, its left/right probes and ResumeByLastEntry read
every probed record through readNeedleAt, which parsed the whole body and
verified the data checksum, although they use only the id, cookie, size and
append time. One record with damaged data on the search path, even a needle
deleted long ago, failed the search and with it the tail sender, a
timestamp-based incremental copy and the volume.move catch-up.

readNeedleAt now reads the header and the trailer and skips the data. The
data checksum is still verified where the data is read, including the tail
receiver for every record it gets. The Rust volume server already parses
only the metadata on this path and does not verify the data checksum.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-19 15:02:25 -08:00
05427e8108 storage: hold the data-file read lock across the append-time search (#12789)
* storage: red test for the append-time search racing a vacuum commit

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* storage: hold the data-file read lock across the append-time search

Fixes #12785

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* storage: Version() writes the superblock version only when it changes

Version() stored the .vif version into SuperBlock.Version on every call, so
each call was a write that raced with readers of the superblock that do not
take superBlockAccessLock, such as the append-time search and the
compaction setup.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* storage: harden the append-time search race test; add a lock-hold budget test

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* storage: Linux-only lock budget; stop before reading search failures

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
2026-11-19 15:01:47 -08:00
Chris LuandClaude Opus 5.5 f7323f2b63 filer: optional HTTP Basic Auth on the filer HTTP ports (#12778)
* security: add a constant-time HTTP Basic Auth check

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* filer: require HTTP Basic Auth on the HTTP ports when configured

With filer.basic_auth.password set in security.toml (or
WEED_FILER_BASIC_AUTH_PASSWORD), a request on the filer HTTP ports, TUS
included, must carry those credentials or a valid filer JWT, and an
access level with no JWT signing key is no longer open. Valid Basic
credentials grant read and write. 401 responses then carry a Basic
challenge so browsers prompt. The credentials reload on SIGHUP with the
signing keys.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* scaffold: document filer.basic_auth in security.toml

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* filer: enable reloaded Basic Auth before swapping the JWT keys

Reload cleared the old signing keys before storing the new Basic
credentials, so a switch from JWT to Basic Auth briefly left both off
and admitted anonymous requests. Store new credentials first and clear
old ones last.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* filer: refuse Basic credentials on cross-origin browser requests

Browsers resend cached Basic credentials on requests other websites
trigger, such as a form POST with mv.from, and the default
cors.allowed_origins "*" lets them through. Accept Basic credentials
only on requests without an Origin, from the filer's own host, or from
an origin explicitly listed in cors.allowed_origins. A JWT is still
accepted from any origin.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* filer: recheck Basic Auth after reading an empty signing key

A request that loaded Basic Auth as off, then read the signing key after
a JWT-to-Basic Reload had cleared it, was let through anonymously. Open
access now also needs Basic Auth to be off after the key read; with
Reload enabling the new credential before disabling the old one, the
two loads cannot both miss a credential that was on throughout.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* security: compare Basic Auth credentials as MACs under a per-process key

A plain SHA-256 of the password is what password-hash scanners flag;
keying the digests with a random per-process HMAC key keeps the
length-independent constant-time compare without holding one.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* util: read a configuration file without merging into the shared viper

LoadConfiguration merges each read into the shared viper, so a key
removed from the file keeps its old value after a reload.
ReadConfiguration reads the file into a fresh viper with the same search
path and WEED_ environment overrides.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* filer: turn Basic Auth off when security.toml no longer sets it

The shared viper merges each SIGHUP reload into what it held, so
removing [filer.basic_auth] kept the old password valid. Read the
credentials from security.toml alone at startup and on reload; an
unreadable file keeps the current setting.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* filer: keep Basic Auth when security.toml cannot be read

A malformed file at startup dropped a WEED_FILER_BASIC_AUTH_PASSWORD
from the environment, and viper reports an unsearchable config
directory as a missing file, which turned Basic Auth off on reload.
ReadConfiguration now always returns the environment overrides and
reports whether the file was read; a reload that could not read it
keeps the current credentials.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* filer: keep reloading Basic Auth from the security.toml first read

A search directory that turned unsearchable made viper fall through to a
lower-priority security.toml, and a reload from a file without
[filer.basic_auth] turned Basic Auth off. ReadConfiguration now takes the
file to read, and the filer reads the file it first found on every reload.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-19 13:19:46 -08:00
f5e091cf91 wdclient, s3: return an embedded S3 gateway to its local filer after failover (#12772)
* wdclient: sort newly discovered filers and log every current-filer switch

Discovery appended new filers in map iteration order, so after the current
filer left, the fallback at index 0 was a random filer. Append them sorted.
Route every change of the current filer through one helper that logs it at
V(0) with its reason (failover, return_home, discovery), so an operator can
see which filer a gateway uses without new metrics.

* wdclient: prefer a home filer and return to it after failover with backoff

A FilerClient with FilerClientOption.HomeFiler set goes back to that filer
once its circuit breaker allows a re-check, instead of staying on whichever
peer it failed over to. Leaving home again soon after a return doubles the
wait before the next return, up to 5 minutes; a return that holds for 10
minutes resets it. Volume lookups and discovery follow the same preference.

* s3: embedded gateway uses its local filer as home filer

weed filer -s3, weed server -s3 and weed mini now build the gateway's filer
address in the form the filer registers in the master (ip:port.grpcPort),
with the gRPC port default applied, and pass it to the gateway as its home
filer, so the gateway returns to its local filer after any failover.
Standalone weed s3 has no home filer and behaves as before.

Refs #12768, follow-up to #12771

* wdclient: switch the current filer only from the filer a request started on

A volume lookup that started on a peer and finished after the client had
returned home moved it back to the peer and counted that as leaving home,
doubling the return backoff although home never failed. Concurrent
failovers could also each count as a separate departure.

Failovers now name the filer they started on and take effect only while it
is still current, via compare-and-swap, so only the winning switch logs and
feeds the backoff. S3's failover walk uses the same rule.

* wdclient: switch the current filer and record the return backoff in one step

The index moved by compare-and-swap before the switch was recorded, and
both ran under the shared read lock. A request reading the current filer
in that gap saw the peer with the old eligibility time and, with home's
breaker still closed after one failure, switched straight back home.
The eligibility check, the move and the bookkeeping now all run under
homeMu.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* wdclient: discovery falls back past a home that is still backing off

When discovery removed the current filer and home was not yet eligible,
the client took index 0, which is home whenever it is listed first. With
home's breaker still closed after a single failure, requests went back to
the filer they had just left. Take the next filer instead.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* wdclient: start return timing on every switch back to the home filer

Only a return_home switch stamped lastReturnHome. Discovery or a peer's
failover could also put the client back on home, and a failure soon
after such a return kept the backoff at its base value instead of
doubling it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Konstantin Lebedev <whitefox@mayflower.work>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-19 13:18:57 -08:00
Chris LuandClaude Opus 5.5 c9f3a02b94 filer: recover a leveldb store whose table a power loss damaged (#12780)
* filer: recover leveldb without rebuilding broken tables

leveldb.Recover rebuilds a table with damaged blocks using the user
comparer and filter on internal keys. The shortened index keys it writes
are not valid internal keys, so a Get on the rebuilt table panics, and
the bloom filter it writes misses every lookup. Recover with a comparer
that keeps full index keys (same name, so the manifest still matches)
and no filter.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* filer: recover a leveldb store when a compaction finds a corrupted table

When a compaction reads a damaged table block, as a power loss can
leave one, goleveldb latches the corruption error and returns it for
every later write until the database is reopened. A plain reopen loads
the same table and the next compaction fails again, so the store kept
rejecting inserts even across restarts.

Treat a corruption error from a write like a storage error, and reopen
through leveldb.Recover, which rebuilds the table without the damaged
blocks, then retry the write.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* filer: rebuild damaged leveldb tables before Recover

leveldb.Recover's own table rebuild orders internal keys with the user
comparer, so a damaged table holding several versions of one key fails
with "keys are not in increasing order" and the recovery gives up.
Overriding the comparer cannot fix that, because the same comparer
also serves the recovered database's user keys.

Rebuild every table that has corrupted blocks first, using an internal
key comparer and full index keys, then let Recover rebuild the manifest
from tables it no longer needs to rewrite. Seed the test tables from a
journal flush so they keep several versions of a key, and check keys
the write test does not rewrite.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* filer: leave unreadable leveldb tables for Recover to drop

table.NewReader reports a truncated table or a damaged footer through
the reader, not its error, so rebuildDamagedTables never stopped on one,
but it rewrote it as an empty table first. Skip tables with nothing
readable and let Recover drop them, and test an unreadable table next
to a healthy one.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* filer: let StrictRecovery drop damaged leveldb tables

With StrictRecovery set, leveldb.Recover drops a table that has corrupted
blocks. Salvaging the readable blocks first would hide the damage from it,
so skip the rebuild under that option.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* filer: hold the leveldb storage lock while rebuilding tables

The table rebuild ran after the failed open released the storage lock
and before Recover took it, so it could rewrite the tables of a database
another caller had open on the same storage. Take the lock for the
rebuild; Recover takes it again afterwards.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-19 13:13:30 -08:00
6bdcd17ed4 master: reject an assign on a follower as Unavailable (#12779)
* master: reject an assign on a follower as Unavailable

Assign on a master that is not the raft leader returned
raft.NotLeaderError as a plain error, which reaches gRPC clients as
codes.Unknown. operation.Assign retries only Unavailable,
ResourceExhausted, Canceled and DeadlineExceeded, so every assign made
during a leader election failed at once instead of waiting for the new
leader.

Return it as codes.Unavailable, like the warming-up status in the same
file, so clients retry within their 30 s assign budget. The message is
unchanged, so callers that match "Not current leader" still work.
StreamAssign now answers such a request in-band with the error message
instead of ending the stream, as it already does for warm-up.

TestAssignOnFollowerIsUnavailable covers it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* pb: keep the cached channel when the peer answers Unavailable with details

gRPC's own Unavailable (dial failure, closing transport) never carries
details, so one that does came from the peer over a working channel.
Closing the shared ClientConn for it cancels every other in-flight RPC
on it with "the client connection is closing".

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* master: tag the follower's assign rejection with a NOT_LEADER ErrorInfo

The detail marks it as the master's answer, so clients retry it
without tearing down their cached channel to that master.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
2026-11-19 13:11:40 -08:00
Chris LuandClaude Opus 5.5 729b872f01 test: let the merge-read FUSE test wait out the master's vacuum drain (#12777)
* test: let the merge-read vacuum outlast the master's write drain

Before compacting a volume the master waits up to 30 s for writes it
assigned to that volume to drain. The test cut each /vol/vacuum call at
30 s and gave the vacuum 60 s, so one or two slow drains failed it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test: retry the merge-read vacuum until the master runs one

A 409 means another vacuum holds the master's lock; check the chunks
only after a vacuum this test asked for has finished.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-19 12:44:48 -08:00
Chris LuandClaude Opus 5.5 972f85159e util/http: probe dead replicas in the background and key no-answer dials by machine (#12776)
* util/http: probe an unanswering server in the background, not on a read

Once the retry interval passed, the first read put the dead server first
and waited out the full dial timeout before trying a live replica. Keep it
last and dial it from a background probe with a short timeout instead.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* util/http: try every server of a machine that does not answer last

Unreachability was kept per host:port, so a powered-off node running one
volume server per disk cost one stalled read per server before each was
known dead. A dial that times out or finds no route now marks the whole
machine; a refused connection still marks only that port and shows the
machine is up.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* util/http: probe an unanswering server with an HTTP request

A bare TCP connect cleared servers that accept connections but never send
response headers, and could not dial an address without an explicit port.
Probe with a HEAD request through the global client instead, and have the
tests wait for the probe to settle.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* util/http: keep a timed-out port last after a sibling port answers

A dial timeout stored only the machine verdict, so a success on another
port of that machine cleared it and the port that timed out went back to
the front. Record the port as well as the machine.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* util/http: classify refused and unroutable dials with Winsock errnos on Windows

Windows reports these as WSAECONNREFUSED, WSAEHOSTUNREACH and friends,
which errors.Is does not match against the syscall constants. Test the
refusal with a real dial to a closed port.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* util/http: wait for background probes through a WaitGroup in tests

A refused probe can finish before the test reads the claimed record, so
inferring completion from the record's timestamp could wait out the
deadline on a probe that already settled.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* util/http: key machine verdicts apart from hosts without a port

A host without an explicit port is also its machine name, so a sibling
port answering cleared the timed-out host along with the machine verdict.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* util/http: group a bracketed port-less IPv6 host with its machine

net.SplitHostPort rejects "[2001:db8::9]", so the machine key kept the
brackets and missed the verdict stored for "[2001:db8::9]:9501".

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-19 12:44:10 -08:00
ed7324611c filer: metrics for metadata subscription delivery (lag, holds, ring fall-offs, claim and watermark ages) (#12773)
* filer: metrics for metadata subscription delivery (lag, holds, ring fall-offs, claim and watermark ages)

* filer: measure subscribe delivery lag after the real send using the mutation stamp, and record a watermark hold once per hold

* filer: skip empty progress markers when sampling subscribe delivery lag

* filer: assert subscribe delivery lag is sampled only after the stream send succeeds

* grafana: panels for the metadata subscription delivery metrics

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* filer: record a watermark hold when the read passes the held entry, not when the next hold starts

* filer: use a monotonic clock for the subscribe hold duration

---------

Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-19 12:43:34 -08:00
Chris LuandClaude Opus 5.5 05a1463855 wdclient: keep the embedded S3 gateway on its local filer across discovery (#12771)
* filer: keep an explicit gRPC port in the embedded gateways' filer address

runFiler handed the embedded S3/WebDAV/IAM/SFTP gateways
NewServerAddress(...).String(), and String() drops the .grpcPort part.
With a non-default -port.grpc the gateways then dialed port+10000 and
did not recognize their own filer in the master's list.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* wdclient: match discovered filers by gRPC address

A filer listed as host:port and the same filer discovered as
host:port.grpcPort were compared as raw strings, so a refresh pruned the
configured filer, added every discovered filer as new in map order, and
reset the current filer to whichever came first; a pushed join added it
a second time. Key filers by their gRPC address: a survivor keeps its
health and current position and takes the discovered address form, and
pushed joins and leaves match the same way.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-19 12:42:31 -08:00
c1c95e127f filer: scoped read-only tokens for metadata subscriptions (#12767)
* security: scoped read-only token for filer metadata subscriptions

A token signed with jwt.filer_signing.subscribe.key names the subscribe
methods and the path prefixes its holder may see. mTLS peers and admin
tokens stay unrestricted. Internal subscribers attach an admin token
when both keys are configured.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* filer: enforce subscription token scope on metadata streams

With jwt.filer_signing.subscribe.key set, SubscribeMetadata and
SubscribeLocalMetadata authorize the caller first. A scoped caller gets
a stream that sends only events whose paths are all under its prefixes,
filters batches, and never carries log file refs.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* filer: send an admin token on internal metadata subscriptions

Peer filers, metadata followers and the lifecycle reader attach an
admin token when the subscribe key is configured, so they keep the full
stream without mTLS.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* shell: add fs.meta.token to mint metadata subscription tokens

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* filer: end scoped metadata subscriptions at token expiry

The scope now carries the token's exp. The scoped stream's context ends
at that time and the call returns Unauthenticated, so an open stream no
longer outlives its token.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* filer.sync: sign metadata subscriptions with the source cluster's key

MetadataFollowOption takes a FilerJwtProvider; filer.sync passes the
one loaded from -a.security or -b.security, so clusters with different
admin keys both accept the subscription.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* shell: report fs.meta.token flag errors

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* filer: give scoped metadata subscribers their own client id

A scoped caller could reuse another subscriber's ClientId with a higher
epoch and replace that registration, disconnecting it. Scoped streams now
register under a server-chosen id that no other subscriber holds.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* security: accept only HS256 metadata subscription tokens

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* security: test HS384 metadata subscription token rejection

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* filer: never let a scoped subscriber replace another registration

The server-chosen id can be taken by another subscriber between choosing
and registering it. Registering scoped streams with the lowest epoch makes
addClient refuse them rather than replace the other registration.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* filer: send position markers for events a scoped token withholds

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* filer: test a scoped stream withholding a whole event batch

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* filer: filter SendMsg on scoped metadata streams

The scoped wrappers only overrode Send, so a SendMsg on the embedded
stream would reach the client unfiltered. Route it through the same
filter and refuse any other message type. Ported from #12790.

Co-authored-by: mindaeon <30555587+mindaeon@users.noreply.github.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* filer: test a scoped subscription through the real subscribe loop

Run SubscribeLocalMetadata with a scoped token over a flushed backlog,
the in-memory tail and live events: only in-scope events arrive, sibling,
"..", encoded and cross-boundary rename paths are dropped, and no log
file references are sent although the client asks for them. Without the
subscribe key every event still arrives. Ported from #12790.

Co-authored-by: mindaeon <30555587+mindaeon@users.noreply.github.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* filer: test that the HTTP API refuses metadata subscription tokens

GET, HEAD, PUT, POST and DELETE all refuse a subscription token, also
when the subscribe key equals the read or write key. Ported from #12790.

Co-authored-by: mindaeon <30555587+mindaeon@users.noreply.github.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: mindaeon <30555587+mindaeon@users.noreply.github.com>
2026-11-19 12:41:16 -08:00
Chris LuandClaude Opus 5.5 668ca45029 s3: ListParts returns each part number once, its latest attempt (#12770)
* s3: list each part number once, its latest attempt

Every UploadPart writes its own NNNN_<uuid>.part entry, so a part number
uploaded twice listed twice, superseded ETag included, and a max-parts page
could hold only the stale attempt. ListParts now groups the entries by part
number, reports the attempt sortEntriesByLatestChunk ranks first (the same
order completion uses), and pages over distinct part numbers, reading past a
page boundary that splits one part number's attempts.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3: test ListParts over repeated part attempts

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3: rank an empty part attempt by commit stamp or mtime

An empty UploadPart body stores no chunks, so sortEntriesByLatestChunk gave
it time zero and an empty re-upload always lost to the attempt it replaced.
Attempts now rank by the filer's commit stamp when both carry one, else by
the later of the first chunk's write time and the entry mtime.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3: keep only the newest attempt per part while listing parts

listPartAttempts held every attempt of a part number until the page was
built, so a heavily retried part grew one ListParts request's memory past
what max-parts bounds. It now compares each entry against the newest seen
for its part number with compareLatestPartAttempt, the comparator
sortEntriesByLatestChunk sorts with.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3: order part attempts by commit stamp, then attempt time

Comparing commit stamps only when both attempts carried one made the order
non-transitive across stamped and unstamped attempts, so listing and
completion could pick different attempts depending on input order. The
commit stamp now leads (unstamped entries predate the field) and the
attempt time breaks ties.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3: list part 10000 after the four-digit part numbers

Part names sort as text, so 10000_<uuid>.part lands between 1000.part and
1000_<uuid>.part. A page could end on part 10000 and the next marker then
skipped parts 1000-9999, and a legacy 1000.part could be reported before
the newer attempt of part 1000 was read. List the four-digit part numbers
first and part 10000 in its own prefix listing.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3: test ListParts paging across part 10000

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-19 12:36:27 -08:00
Chris LuandClaude Opus 5.5 dcbea0626f test/s3: patch s3-fast-list's data map race in the listing study build (#12766)
test/s3: build s3-fast-list with its data map draining the last page

The data map task reads the channel and then the list-task state. When
the list task sends its last page and completes between the two reads,
the map is dumped without that page and the tool exits 0, so a one-page
prefix comes back empty. Read the state before the channel.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-19 12:21:32 -08:00
Chris LuandClaude Opus 5.5 de4075660f telemetry: refresh the server module's go.mod so the integration test builds (#12765)
* telemetry: refresh the server module to the root module's dependency versions

The server module replaces seaweedfs with ../.., so root dependency
bumps raised prometheus/common, prometheus/procfs and x/sys past the
versions its go.mod listed, and go build refused to run.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* ci: run the telemetry integration test on root go.mod changes

The telemetry server module builds against the root module through a
replace directive, so a root dependency bump can leave its go.mod stale
without touching telemetry/.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-19 00:55:09 -08:00
Chris LuandClaude Opus 5.5 652133a28a telemetry: keep the newest release in the versions chart (#12764)
The versions chart keeps the 8 versions with the most clusters on the last
day and folds the rest into "other". A new release starts small, so 4.54
(282 clusters) was folded in while 4.53 shrank as clusters upgraded,
which showed as a dip across the named versions. The newest plain release
now always keeps its own band.


Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-19 00:08:19 -08:00
e012fca9a2 mount: keep lookups from stalling behind directory cache rebuilds (#12754)
* mount: release the meta cache lock between batches of a rebuild wipe

deleteFolderChildrenForRebuild held mc's write lock while it listed every
cached child of the directory and deleted them. Every metadata read in the
mount (FindEntry, IsNameFresh, isOversized) takes the read lock, and a
waiting writer blocks new readers, so while a large directory was wiped,
lookups and getattrs in unrelated directories queued behind the scan. The
scan gets slower as leveldb accumulates the deletion markers of earlier
wipes of the same directory, which happens quickly once lookups rebuild
expired directories.

Wipe rebuildWipeBatchSize (1024) children per lock hold and drop the lock
between batches. A child is judged pinned under the same lock that deletes
it, so a local-only create is kept whether it was pinned before the wipe
started or between batches. The listing that refills the directory already
ran outside the lock, so batching adds no interleaving the build did not
allow.

Add LevelDBStore.BatchDeleteEntries, the counterpart of BatchInsertEntries.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>

* mount: bound how long a lookup waits on the directory rebuild it starts

Since #11268 a lookup whose parent's cache expired rebuilds the parent and
every expired ancestor up to the mount root before it answers, and every
lookup beneath them joins the same builds. With a slow listing (a large
directory, a busy filer, or the meta cache lock held by another rebuild) a
plain stat, an ls -l, or the parent touch after mkdir waited for all of it.

Wait at most lookupRebuildTimeout (2s), then fall back to one LookupEntry
RPC. Since #12206 the shared build does not run on the requester's context,
so it keeps going after the waiter gives up and later lookups find the
directory cached. A timed-out wait records a rebuild attempt, so lookups
during the cooldown do not queue on a build that is already known to be
slow.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>

* mount: wait for the rebuild's listing in TestLookupDoesNotWaitOutSlowRebuild

The build runs on its own goroutine, so the lookup that stopped waiting
for it can return before the build's ListEntries reaches the filer. The
test read `started` without waiting right after the lookup returned, which
relied on an ordering nothing guarantees, and the ListEntries count taken
afterwards assumed the same. Wait for the listing to arrive, bounded by
the test's deadline.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>

* mount: keep children inserted while a rebuild wipe runs

A deferred create inserts its placeholder before Create registers the
handle that pins it. The single-lock wipe made such an insert wait until
it finished, so the placeholder survived; a batch that runs in that gap
judged it unpinned and dropped it, and the rebuilt listing then omitted
the file until its flush.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* mount: drop a cached path that an event vacates during a rebuild wipe

A build replays only the events later than its listing's snapshot and
leaves the earlier ones to the listing, which starts after the wipe. With
the wipe in batches a file's whole life fits inside it: created, flushed,
then unlinked or renamed away. Both events are earlier than the snapshot
and are skipped, the listing has no such file, and the placeholder the
wipe kept stays in the published directory until the next rebuild.

While a directory's wipe runs, an event buffered for it now takes the
path it vacates out of the store at once, fenced by versions like any
other event. After the wipe the store holds what the listing put there,
and a removal earlier than the snapshot may still arrive, so from then on
the event is left to the replay as before.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>

* mount: check which file a removal took away during a rebuild wipe

Two things the removal applied during a rebuild wipe got wrong.

A placeholder is unversioned, so the version gate lets every removal
through. One that arrived late, for an earlier file of the same name,
took out the placeholder of a file created since. The entry the event
took away is now compared with the cached one, by inode or, with no
inode to compare, by creation time. An event that only names the file
cannot tell and is taken at its word.

The entry was deleted without OP=MV, so taking one name of a hard-linked
file out of the cache also took one off the count its other names share.
A replayed event that vacates a path already deletes with OP=MV.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>

* mount: keep a removal the rebuild wipe declined out of the replay

The wipe declines a removal when what is cached there is newer than the
event or is another file than the one it took away. The event stayed
buffered anyway. An unversioned listing clears the children's version
records and replays every buffered event, so the declined removal then
deleted the entry the wipe kept and the directory was published without
it.

A declined removal now buffers only the name it puts in place, so a
rename still lands on replay.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* mount: check a declined removal again when the build replays it

The previous commit dropped the removal half of an event the wipe
declined, judged by the file cached when the event arrived. Earlier
buffered events can replace that file during the replay: a listing
attempt that failed leaves its events buffered, so the retry's wipe may
see the first attempt's file while the replay puts a later one in place
first. The dropped delete then left that later file published.

The wipe now records which removals it declined, and the replay checks
each against the entry it finds at that point, replaying only the name
the event puts in place when that entry is still another file. A
removal declined by version is replayed as before.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* mount: keep a declined removal's invalidation off the placeholder's handle

After the replay the build invalidates every buffered event again,
raised to the snapshot, and it still sent the original of a removal the
replay rejected. A late delete of an earlier file then reached the
handle of the placeholder at that name: the placeholder is unversioned,
so the raised timestamp got past it and the handle was marked deleted.

The replay's decision on a declined removal now replaces the event in
the buffer, and the invalidations use what was replayed. It is made
before the snapshot check, so a removal the listing covered is judged
the same way.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* mount: keep an event's signatures on the halves a rename is split into

A declined removal replays as its rename's create half, and a rename across
a building directory is split in two. Both dropped the event's filer
signatures, so a handle stamped by that filer no longer fenced the replayed
invalidation by timestamp and could take older listed contents over newer
ones it had read.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* mount: drain invalidations around the declined-removal build check

Invalidations run on a background worker, so the baseline could miss the
removal's own and the final read could run before the build's arrived.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: MapleWhiteZZZ <18653045+MapleWhiteZZZ@users.noreply.github.com>
Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
2026-11-19 00:04:03 -08:00
Chris LuandClaude Opus 5.5 3159b8408c filer, volume: fsync chunk deletes under an fsync path rule (#12759)
* volume: fsync a batch delete that asks for it

BatchDeleteRequest gets an fsync field, and the Go and Rust volume
servers pass it to the needle delete, which already orders the
tombstone, .dat sync, publish and .idx sync for a durable delete.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* operation: pass an fsync flag to batch deletes

DeleteFileIdsWithLookupVolumeId and DeleteFileIdsAtOneVolumeServer take
an fsync argument and send it in the BatchDeleteRequest. Every caller
passes false for now.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* filer: fsync chunk deletes under an fsync path rule

DeleteChunks passes the path rule's fsync flag to the deletion queue.
The flag is kept with the pending record until the delete is
confirmed, so retries and re-queues keep it, and the batch and retry
loops send marked ids in a durable BatchDelete.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* filer: keep fsync marks in the deletion ledger

Marked ids go into the trailing parts of a manifest ledger, counted by
fsyncParts. Older filers read those parts as plain ids and still delete
them; a reload or a claimed foreign ledger restores the marks. A ledger
without marks keeps the single array form.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* filer: record a queued deletion before the delete loop can see it

queueDeletions published the ids to the hot queue before taking the
ledger lock, so a concurrent flush could delete an id before its fsync
mark landed and then forget the mark with the record.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* filer: fsync a folder delete's chunks by each child's path rule

A recursive delete collected its file children's chunks and resolved
the rule once for the folder, which misses a trailing-slash rule on
the folder itself and file-prefix rules. Each file child's own rule now
decides fsync; the folder rule still decides chunk deletion.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* filer: keep a newer fsync request when a plain delete finishes

An id re-queued with fsync while a plain delete of it was in flight
lost its mark when that delete succeeded, so the queued copy also ran
without fsync. A plain attempt now leaves a marked record for its
queued copy; a durable delete of an already-deleted needle still syncs
the tombstone.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* filer: delete a folder's plain child chunks without fsync

Children whose own rule turns fsync off were still deleted durably when
the folder's rule asked for fsync. Only the children marked by their
own rule now go out with fsync.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* filer: retry a durable chunk delete the volume server failed

A volume server rolls back a durable delete it cannot sync and answers
with a 5xx carrying the I/O error text, which the classifier took as
permanent, so the record and its fsync mark were dropped while the
needle stayed live. A 5xx answer to a durable delete is now retryable,
within the usual attempt limit.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-19 00:02:43 -08:00
Chris LuandClaude Opus 5.5 f88751c3ff s3api: evaluate request condition keys for STS sessions (#12762)
* s3api: build request condition keys in one helper

Split the keys ExtractConditionValuesFromRequest reads from the request
itself into RequestConditionValues, keeping aws:SourceIp and the
bucket-listing keys in the method.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3api: evaluate request condition keys for STS sessions

The IAM integration context for STS and OIDC sessions carried only the
source IP, legacy user agent/referer keys and claims, so role and session
policy conditions on s3:x-amz-*, aws:SecureTransport, aws:CurrentTime,
s3:RequestObjectTag/* and the other request keys never matched: a Deny
on them never fired, and a negated Deny always did. Add the same request
keys the static identity path evaluates.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3api: report SigV4 presigned requests as REST-QUERY-STRING

s3:authType recognized only SigV2 query authentication, so a policy
condition on REST-QUERY-STRING never matched a SigV4 presigned URL.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-19 00:01:32 -08:00
Chris LuandClaude Opus 5.5 2089abecf3 ci: cache Snowflake suite Maven deps and fall back to a Central mirror (#12763)
* ci: cache the Snowflake suite's Maven dependencies under their own key

setup-java's maven cache key hashes the repository pom.xml files, which
other Java workflows share. The Snowflake job restored that cache, which
lacks the suite's plugins and dependencies, and never saved its own because
the key already existed. Every run downloaded the whole suite dependency
tree from Maven Central and failed when Central answered 429.

Key the cache on run.sh, which pins the suite revision.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* ci: fall back to Google's Maven Central mirror in the Snowflake job

A cold cache still resolves about 36 MB of plugins and dependencies, and
Maven gives up after three 429 retries within 30 s. Add the mirror as a
settings-profile repository; Maven tries it first and moves on to Central
for any artifact the mirror fails to serve.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-19 00:00:41 -08:00
Carlos Leyva 6d0a6e0935 filer: mint read_auth for the data chunks behind manifest chunks (#12756)
* filer: mint read_auth for the data chunks behind manifest chunks

* filer: cover every chunk behind manifests in read_auth (data + nested), minted after resolution, and skip when read signing is off

* filer: pass the master client as manifest cache invalidator, and move the read_auth helpers below populateBatchReadAuth for the enterprise merge
2026-11-18 19:37:06 -08:00
Chris LuandClaude Opus 5.5 9b94f00325 test: mount pinned Iceberg jars into the Spark catalog tests (#12757)
* testutil: fetch pinned Maven Central jars with a mirror fallback

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test: mount pinned Iceberg jars into the Spark catalog containers

Each test started a new Spark container that resolved the Iceberg runtime
and AWS bundle through Ivy from Maven Central, so a run downloaded them six
times and a single refused request left Spark retrying until the package
timed out.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* ci: cache the Iceberg jars for the Spark catalog tests

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* testutil: test the verified jar download

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test: bound Maven jar downloads by one deadline shared across retries

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test: give each Maven mirror attempt its own timeout within the download budget

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-18 19:35:54 -08:00
Chris LuandClaude Opus 5.5 d2a0002869 s3api: fix listing regressions from argument validation (empty continuation-token, V1 Prefix encoding) (#12760)
* s3api: accept an empty ListObjectsV2 continuation-token

An empty continuation-token lists from the start and is echoed back
empty, as S3 does; rejecting it broke s3-tests
test_bucket_listv2_continuationtoken_empty.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3api: leave the ListObjects V1 Prefix unencoded under encoding-type=url

S3 encodes the V1 Delimiter, markers and keys but echoes Prefix as sent,
and botocore, which always asks for url encoding, decodes the V1 Prefix
only for V2. Encoding it broke s3-tests test_bucket_list_prefix_unreadable.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3api: use a codespell-clean key in the versions encoding test

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3api: URL-encode a V1 Prefix that XML cannot carry

Leaving the V1 Prefix raw under encoding-type=url turned a control
character such as \x01 into U+FFFD in the response. Such a prefix is
URL-encoded again; prefixes that are valid XML text, including '\n',
stay as sent.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-18 19:34:57 -08:00
github-actions[bot] c01e87689e docs: regenerate star history chart 2026-11-19 02:58:55 +00:00
Chris LuandClaude Opus 5.5 7b4bdb77bb master: export writable volumes per disk and reset leader-only layout gauges on followers (#12751)
* master: export writable volumes per disk for each volume layout

The layout gauges only report how many volumes of a collection take
writes, not where they sit, so two writable volumes sharing one disk
cannot be seen or alerted on. The leader now exports
writable_volumes_per_disk with the layout labels plus data center, rack,
node and disk id, counting the replicas of the layout's writable volumes
on each disk (state=writable) and those of them that are crowded
(state=crowded). Series for disks that no longer hold a writable volume of
the layout, and for layouts that are gone, are deleted.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* master: reset the leader-only layout gauges on followers

The layout writable, crowded, under-replicated and per-disk gauges are only
set by the leader, so a master that lost leadership kept exporting the
values of its last term indefinitely, and dashboards that aggregate over
masters could show numbers days old. A master that is not the leader now
clears them on each periodic pass.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* grafana: chart disks holding several writable volumes of a collection

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* grafana: keep disks of different clusters apart in the per-disk panel

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-18 16:17:04 -08:00
Chris LuandClaude Opus 5.5 1fd5008d63 s3api: listing argument validation and encoding-type=url for versions, prefix and delimiter (#12753)
* s3api: reject an empty ListObjectsV2 continuation-token

AWS answers continuation-token= with 400 InvalidArgument ("The continuation
token provided is incorrect"); SeaweedFS listed from the start of the bucket.
No token this server issues is empty, and raw-key and base64: tokens keep
resuming as before.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* s3api: reject an encoding-type other than url on listings

AWS answers any encoding-type other than url with 400 InvalidArgument
("Invalid Encoding Method specified in Request") on ListObjects,
ListObjectsV2, ListObjectVersions, ListMultipartUploads and ListParts;
SeaweedFS ignored the value and listed unencoded keys.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* s3api: apply encoding-type=url to ListObjectVersions

The handler accepted encoding-type=url but returned raw keys. It now
URL-encodes Key in Version and DeleteMarker entries, CommonPrefixes, Prefix,
Delimiter, KeyMarker and NextKeyMarker with the urlPathEscape form ListObjects
uses, and echoes EncodingType.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* s3api: URL-encode Prefix and Delimiter in ListObjects responses

Under encoding-type=url, ListObjects and ListObjectsV2 encoded keys,
CommonPrefixes, markers and StartAfter but echoed Prefix and Delimiter raw,
so a client decoding them once corrupted a prefix holding '%' or '+'.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-18 15:55:41 -08:00
646549eee1 volume growth: spread writable volumes of a layout over servers and disks (#12749)
* volume growth: spread writable volumes of a layout over servers and disks

Volume growth picked every server, and the volume server picked every disk,
independently: the server weighted by free slots, the disk by the lowest
total volume count. A grow request therefore stacks several writable volumes
of one collection on some disks while most disks get none (a Poisson spread:
with 160 replicas on 479 disks about 20 disks get two or three), and their
writes share those disks.

Data centers and racks are still picked weighted by free slots. Inside a
rack the server is now one of those holding the fewest writable volumes of
the layout per disk, weighted by free slots among them; this covers the main
server, same-rack replicas and replicas walked into other racks and data
centers. On the volume server the disk is the one with the fewest writable
volumes of the collection, then the lowest volume count as before.

A grow request registers each volume before placing the next, so a request
smaller than the number of disks lands on distinct disks.

Fixes #12746

* volume growth: scan the layout load once per grow request

Placing each volume rescanned all writable volumes of the layout under its
read lock. RecordAssign takes that lock exclusively on every assign, so with
thousands of writables (0.8 ms for 10000) every volume of a grow request
stalled assigns once more.

GrowByCountAndType now scans once and findAndGrow adds every volume it
creates to the counts. Per volume, placement costs about 130-160 us on 480
servers whatever the number of writables, against 120 us before the spread
change; the single scan is 0.17 ms for 2000 writables and 0.8 ms for 10000.

* volume growth: address review of the spread placement

- Count only disks that report volume slots in a server's load. An
  unavailable disk, or one low on space with no volumes, is still listed in
  DiskTags with no slots; counting it understated the load of a server with
  failed disks and piled volumes onto its remaining ones.
- On the volume server, judge a volume full by its .dat size, which the
  master compares with the limit, not by ContentSize.
- A server that refuses an allocation ranks last for the rest of the grow
  request, and the request retries the volume up to 3 times. The least
  loaded server is picked first deterministically, so without this one server
  that cannot fit a preallocated volume stopped the layout from growing.
- Scan the layout load after warmup, so a grow request issued right after a
  leader change sees the writable volumes the returning servers report.

* volume growth: keep the request-local placement load ahead of the request fields

The JSON request fields stay together at the end of VolumeGrowOption, and
the comment now says the load lives for the whole grow request.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* volume(rs): place a new volume on the disk with the fewest writable volumes of its collection

Store::add_volume picked the disk with the most free slots, so the Rust
volume server ignored the collection spread rule the Go server now applies.
find_free_location ranks disks by the writable volumes of the collection
first (not read-only, .dat below the volume size limit), then by free slots.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* volume growth: leave a refusing server out of every placement tier

A server that refused an allocation only ranked last among the servers of
its rack, so the weighted data center and rack picks could still choose its
rack on every retry while other racks had room. It now offers no free slots
for the rest of the grow request, which takes its slots out of the rack and
data center weights too. If only refusing servers are left, they are tried
again.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* volume growth: require the full same-rack count only in the main rack

The data center check asked every rack for SameRackCount+1 free servers,
but only the main rack holds the same-rack copies; each other rack takes one.
With a refused server left out of its rack, the check rejected a feasible
placement, the retry put the refused server back, and it was picked again.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Konstantin Lebedev <whitefox@mayflower.work>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-18 15:51:13 -08:00
Chris LuandClaude Opus 5.5 9cd183b04f volume: fsync a needle delete that asks for it (#12736)
* volume: fsync a needle delete that asks for it

A delete now takes fsync like a write: the tombstone is synced before the
needle map drops the needle, and the .idx row before the delete is
acknowledged. A failed .dat sync takes the tombstone back off the .dat.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* volume: honor fsync=true on DELETE and forward it to replicas

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* volume(rs): fsync a needle delete that asks for it

Mirrors the Go volume server: DELETE with fsync=true flushes the tombstone
before the index drops the needle and the .idx before the ack, and a failed
.dat flush takes the tombstone back off the .dat.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* volume: let a durable delete through on a full volume

A queued delete hit the batch worker's volume size check, which a
tombstone append is exempt from inline, so fsync=true deletes failed once
the volume reached its hard limit.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* volume: sync a durable delete of an already-deleted needle

An earlier delete without fsync may have dropped the needle with its
tombstone still unsynced, so the inline path syncs before acking anyway,
as the batch worker already does.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* volume: refuse a batch request that races the worker stopping

stopWorker can close the batch channel between startWorker handing it out
and the send, which panicked. The request now falls back to the inline path.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* volume(rs): sync a durable delete of an already-deleted needle

Mirrors the Go volume server: an earlier delete without fsync may have
dropped the needle with its tombstone still unsynced.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* volume(rs): run a durable delete on a blocking thread

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* volume: keep syncDelete's signature for existing callers

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* volume: refuse a write a delete cancelled before it appends

A write cancelled while still queued behind the volume locks used to
append, publish and then tombstone itself without a sync, after its
delete may already have been acknowledged as durable. A crash could
keep the row and lose the tombstone. Refuse it before the append.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* volume(rs): refuse a write a delete cancelled before it appends

Mirrors the Go change: a write cancelled while still queued behind the
volume locks no longer appends, publishes and tombstones itself without
a sync after its delete may already be durable.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* volume(rs): drop the write guards for a durable delete's flushes

The durable delete held the write lease and the io write guard across
both fsyncs, so reads of the volume waited on them. Append and publish
the tombstone under the guards and flush with only the io read side, as
write_synced_run and the Go path already do. do_delete_request keeps its
signature.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* volume: stop taking writes when a durable delete fails to index

The needle map drops the needle before it appends the .idx row, so after
a failed append a durable retry found nothing to delete and acked, and a
later write could bury the unindexed tombstone so a reload brought the
needle back. Fence the volume the way a failed durable write does.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* volume(rs): stop taking writes when a durable delete fails to index

Rust counterpart of the Go change: the needle map drops the needle
before it appends the .idx row, so a failed append left a durable retry
nothing to delete, and a later write could bury the unindexed tombstone.
publish_delete_request now fences the volume for a durable delete, as
publish_write does.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* volume: stop taking writes when any delete fails to index

A non-durable delete whose .idx append failed left the volume writable with
the needle already dropped from the map, so a later durable retry found
nothing to delete and acknowledged without the missing row.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* volume(rs): stop taking writes when any delete fails to index

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* volume: take the fsync flag on the existing delete functions

DeleteVolumeNeedle, deleteNeedle2 and syncDelete take fsync instead of
gaining *WithFsync variants.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* volume(rs): take the fsync flag on the existing delete functions

delete_volume_needle and delete_needle take fsync instead of gaining
*_with_fsync variants; the durable path and the deletable checks fold
back into delete_needle, and the cancelled-write check mirrors Go's
isCancelledWrite.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-18 15:18:29 -08:00
Chris LuandClaude Opus 5.5 23eef59417 test/s3: run the s3-listing-study listing tools and strategies in CI (#12750)
* test(s3): s3-listing-study integration suite (WIP)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* test(s3): run the s3-listing-study tools against SeaweedFS (WIP)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* test(s3): s3-listing-study deleted-prefix probe, fresh data dir per run

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* test(s3): listing study ports keep clear of every gRPC companion

free_ports.py compared only base ports, so a base port could land on
another server's base+10000 gRPC listener.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test(s3): listing study ports stay below the ephemeral range

Base ports of 20000-55000 put both them and their gRPC companions inside
32768-60999, so an outgoing connection made while weed server starts could
take a port before the server that was given it binds.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test(s3): listing study checks the order inside each listing page

parse_page sorted Contents and CommonPrefixes together before the probes
compared them, so keys returned out of order within one page passed. Check
each list's byte order first; the versions pager does the same.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test(s3): listing study applies --only before running a probe

The filter was applied to the probe results, so a run limited to one tool
still sent every reference listing request and the deleted-prefix check.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test(s3): listing study requires CommonPrefixes from shallow modes

The CommonPrefixes comparison ran only when a tool returned at least one
prefix, so a delimiter listing of the canary root that came back empty
passed. Compare them for every shallow mode except the two whose
normalizer keeps keys only.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-18 15:17:25 -08:00
Chris LuandClaude Opus 5.5 04c7b12238 s3api: reject unknown notification elements; fail test-with-server on a stuck shutdown (#12752)
* s3api: reject unknown elements in PutBucketNotificationConfiguration

xml.Unmarshal skips children that match no field, so a configuration
whose only child was an unmapped element parsed as empty and returned
success while nothing was stored.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test/s3/tigris_compat: fail test-with-server when the server does not stop

A passing suite returned success even when stop-server gave up with the
server still running.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-18 15:15:38 -08:00
Chris LuandClaude Opus 5.5 f5418845ce test/s3: run tigrisdata-community s3-api-compat-tests in CI (#12741)
* test/s3: run tigrisdata-community s3-api-compat-tests in CI

Runs the suite, pinned to a commit, against weed mini through a small
provider patch, and fails on any test that is not in
expected_failures.txt or on a listed test that starts passing.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* test/s3/tigris_compat: prepare the suite again when the patch changes

The checkout was reused whenever it sat at the pinned commit, so an
edited seaweedfs-provider.patch, or a checkout whose patch failed to
apply, kept being built as it was. Record the commit and the patch's
hash once the patch applies, and prepare the checkout again when either
differs.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test/s3/tigris_compat: keep the pid file of a server already running

start-server removed weed-server.pid before it found the port taken, so
a second start lost the pid of the server holding it and stop-server
could no longer stop that one. Remove the file only once the port is
free, and have test-with-server stop only a server it started.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test/s3/tigris_compat: bound the S3 port probes

A listener that accepts but never answers held curl, and with it the
readiness loop and the cleanup after it, for good. Give each probe two
seconds.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test/s3/tigris_compat: expect a listed test to end with its listed status

A listed test passed the check with any non-pass status, so a setup error
that skipped the call under test (RestoreObject's PutObject, say) went
unnoticed. Entries now expect fail unless they name another status; only
GetBucketAccelerateConfiguration, whose setup PUT is unsupported, expects
error.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test/s3/tigris_compat: refuse to start while the recorded server still runs

A second start-server before the first server answered on the S3 port
passed the port probe and dropped the pid file of the live server, so
stop-server could no longer stop it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test/s3/tigris_compat: recognize the test server by its command line

start-server and stop-server took any process named weed at the recorded
pid for the test server, so a weed process that reused the pid would be
reported as running and killed by stop-server. Match the command line
start-server launches instead.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test/s3/tigris_compat: recognize the test server by its start time

A weed mini started from another checkout has the same command line, so
a pid it reused still matched. Record the process start time next to the
pid at launch and require both before treating the process as the test
server.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test/s3/tigris_compat: PutObjectTagging_VersionId passes now

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test/s3/tigris_compat: keep the server records until the server exits

stop-server waited a fixed two seconds after SIGTERM and then removed
weed-server.pid and weed-server.started even if weed was still shutting
down, so a later stop-server could not find it and clean could delete
data it still used. Wait up to 30 seconds for the recorded server to exit
and keep the records, failing, while it still runs.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test/s3/tigris_compat: start the server on an empty data directory

A data directory left by an earlier run on other ports made the master
freeze volume growth (topology id mismatch), and every PutObject failed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-18 15:14:52 -08:00
Chris LuandClaude Opus 5.5 53bc62c773 s3api: accept the accelerate and notification configurations the bucket already reports (#12747)
* test/s3: run tigrisdata-community s3-api-compat-tests in CI

Runs the suite, pinned to a commit, against weed mini through a small
provider patch, and fails on any test that is not in
expected_failures.txt or on a listed test that starts passing.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* test/s3/tigris_compat: prepare the suite again when the patch changes

The checkout was reused whenever it sat at the pinned commit, so an
edited seaweedfs-provider.patch, or a checkout whose patch failed to
apply, kept being built as it was. Record the commit and the patch's
hash once the patch applies, and prepare the checkout again when either
differs.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test/s3/tigris_compat: keep the pid file of a server already running

start-server removed weed-server.pid before it found the port taken, so
a second start lost the pid of the server holding it and stop-server
could no longer stop that one. Remove the file only once the port is
free, and have test-with-server stop only a server it started.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test/s3/tigris_compat: bound the S3 port probes

A listener that accepts but never answers held curl, and with it the
readiness loop and the cleanup after it, for good. Give each probe two
seconds.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3api: accept PutBucketAccelerateConfiguration with Status=Suspended

GetBucketAccelerateConfiguration always reports Suspended, but the PUT
answered 501 for every request. Suspended is now accepted, Enabled
stays NotImplemented, and the accelerate subresource resolves to
s3:GetAccelerateConfiguration / s3:PutAccelerateConfiguration.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* s3api: accept an empty PutBucketNotificationConfiguration

GetBucketNotificationConfiguration always reports an empty
configuration, but the PUT answered 501, including for the empty
configuration clients send to clear notifications. An empty one is now
accepted; a topic, queue, Lambda or EventBridge destination stays
NotImplemented.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* s3api: reject accelerate and notification bodies with the wrong root

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* test/s3/tigris_compat: expect a listed test to end with its listed status

A listed test passed the check with any non-pass status, so a setup error
that skipped the call under test (RestoreObject's PutObject, say) went
unnoticed. Entries now expect fail unless they name another status; only
GetBucketAccelerateConfiguration, whose setup PUT is unsupported, expects
error.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test/s3/tigris_compat: refuse to start while the recorded server still runs

A second start-server before the first server answered on the S3 port
passed the port probe and dropped the pid file of the live server, so
stop-server could no longer stop it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test/s3/tigris_compat: recognize the test server by its command line

start-server and stop-server took any process named weed at the recorded
pid for the test server, so a weed process that reused the pid would be
reported as running and killed by stop-server. Match the command line
start-server launches instead.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test/s3/tigris_compat: recognize the test server by its start time

A weed mini started from another checkout has the same command line, so
a pid it reused still matched. Record the process start time next to the
pid at launch and require both before treating the process as the test
server.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test/s3/tigris_compat: PutObjectTagging_VersionId passes now

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test/s3/tigris_compat: keep the server records until the server exits

stop-server waited a fixed two seconds after SIGTERM and then removed
weed-server.pid and weed-server.started even if weed was still shutting
down, so a later stop-server could not find it and clean could delete
data it still used. Wait up to 30 seconds for the recorded server to exit
and keep the records, failing, while it still runs.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test/s3/tigris_compat: start the server on an empty data directory

A data directory left by an earlier run on other ports made the master
freeze volume growth (topology id mismatch), and every PutObject failed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-18 14:55:18 -08:00
Chris LuandClaude Opus 5.5 1f5cfc7552 s3api: return x-amz-version-id from object tagging on versioned buckets (#12745)
* s3api: return x-amz-version-id from object tagging on versioned buckets

PutObjectTagging, GetObjectTagging and DeleteObjectTagging now report
the version they acted on, "null" for an object written before
versioning was enabled.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* s3api: set the tagging x-amz-version-id only on success

The header was set before the tag-conditioned policy recheck, so a
denied request carried the version id on its AccessDenied response.
Also test the "null" version and require distinct version ids.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-18 14:30:55 -08:00
Chris LuandClaude Opus 5.5 b1c315b837 s3api: report no ownership controls after DeleteBucketOwnershipControls (#12748)
* test/s3: run tigrisdata-community s3-api-compat-tests in CI

Runs the suite, pinned to a commit, against weed mini through a small
provider patch, and fails on any test that is not in
expected_failures.txt or on a listed test that starts passing.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* s3api: report no ownership controls after DeleteBucketOwnershipControls

GetBucketOwnershipControls reports BucketOwnerEnforced for a bucket that
never stored a setting, so it kept reporting it after the setting was
deleted. The delete now records that it happened, and GET answers
OwnershipControlsNotFoundError until ownership is put again. How
objects are written after the delete is unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* s3api: answer DeleteBucketOwnershipControls with 204 No Content

AWS answers a successful DeleteBucketOwnershipControls with 204 and no
body; this answered 200 with an empty OwnershipControls document.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-18 14:25:15 -08:00
aceb089bc8 lance: per-table cleanup retention and on-demand purge of deleted rows (#12739)
* lance: per-table cleanup retention and on-demand purge of deleted rows

A vector deleted from an index stayed in storage until compaction
rewrote its fragment (only above 10% deletions) and the cleanup job's
168 h retention, set once per worker, aged out every version that
referenced it.

Let a table override retain_hours and min_versions in its Lance config
(seaweedfs.cleanup.retain_hours, seaweedfs.cleanup.min_versions), and
let it request a purge (seaweedfs.purge.requested). The cleanup job
proposes a table with a pending purge regardless of the version floor
and then rewrites every fragment with deletions, retrains the indices
(a remapped IVF_FLAT index still holds the deleted vector), keeps only
the latest version, and records the request as served. A version pinned
by a tag fails the purge and leaves the request pending.

The test scans every file of a local table for a deleted vector's key,
metadata and values, and finds none after the purge.

* lance: refuse to purge a sharded vector index

A retrain builds every index name over the whole table, so the purge
turned shards of one vector index (one name per fragment set) into
names that each covered every fragment, indexing every row once per
shard. Check before compacting and fail with the request left pending,
and retrain only the vector indices.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* lance: rebuild scalar indices in a purge

Compaction only remaps scalar indices, and a retrain leaves them alone,
so a purge reported success while a bitmap index still listed the
deleted row's value, and under stable row ids a BTree index still held
the deleted key. Build each scalar index again from the remaining rows
with the parameters it was built with.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* lance: purge through the cleanup job's retention policy

The purge built its own cleanup policy, so a deployment that has the
cleanup job keep more than the floor and the window (versions an
unleased reader may still be on) would have lost them to a purge. Clean
up through retention_policy, the one the job itself uses, with the
latest version kept.

Mark the request served only once the oldest version left is one the
rebuild wrote, and record the rebuild so a purge that has to wait for
its cleanup (a tag, or that policy) retries only the cleanup instead of
compacting and retraining the indices on every pass.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* vector: lease the version every list page reads

A list page read its version without a lease unless it was a segment
that read data and metadata twice. A purge removes every version but
the latest, and only a lease tag stops it, so a listing that opened the
version before the purge's compaction could lose its files mid-read.
Pin it as get_vectors and query_vectors do.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* lance: give the cleanup job a build's execution time

A purge compacts the table and retrains its indices inside a cleanup
job, but the job declared no runtime defaults, so the admin held it to
its 90 s default and could expire a purge mid-rebuild. Declare the same
budget the indices job uses for a retrain.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-18 14:24:28 -08:00
c7d8b106d5 mount: follow a peer unlink to the surviving hard link for closed inodes (#12737)
* mount: drop a peer-unlinked name only while its inode keeps another

Move the survivor check from followSurvivingLink into InodeToPath.RemoveAlias
so the check and the removal happen under one table lock.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* mount: follow a peer unlink to the surviving hard link for closed inodes

A peer deleting one name of a hard link only moved an open handle to the
surviving name. A closed inode kept the deleted name as its primary path, so
the next open by the kernel's nodeid failed with ENOENT.

Co-authored-by: skymazer <54621405+skymazer@users.noreply.github.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* mount: share the cached-entry freshness check of peer invalidations

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* mount: keep a name a local rename reclaimed before the peer unlink arrived

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* mount: keep a name a local rename or link gave the inode after an older peer delete

The cached-entry check only covers a cached parent. InodeToPath now records,
per hard-linked inode, the newest name a local rename or link gave it with
the filer stamp, and the closed-inode peer-delete path keeps that name when
the delete is older. The record lives in a side map dropped on Forget, so
InodeEntry keeps its size class.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* filer: sign rename stream responses with the filer signature

A rename response carries the stamp of the filer that applied it, so the mount
can tell which clock the stamp belongs to, as it does for create acks.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* mount: order a peer delete against a local rename or link only on one filer clock

The stamp of a local rename or link comes from the filer that applied it, and
a delete logged by another filer runs on a different clock. Keep the name only
when both carry the same nonzero filer signature; otherwise drop it as before.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* mount: drop the local-name stamp fence for uncached parents

Ordering a delayed peer delete against a local rename by filer stamp needs a
record per alias, a shared filer clock, and the open-handle path to follow;
that is a larger change than this fix. Back to the cached-entry check, which
covers a cached parent.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: skymazer <54621405+skymazer@users.noreply.github.com>
2026-11-18 14:13:40 -08:00
a3b2fd722d mount: report opens for reading with TouchAccessTime (-accessReport) (#12721)
* mount: report opens for reading with TouchAccessTime (-accessReport)

An open answered from the metadata cache never reaches the filer, so a
filer that tracks access cannot see that a cached file is still being
read. With -accessReport, the mount sends one TouchAccessTime per open
for reading, with the open time, whether the open was answered from the
cache or not. Write-only opens are not reported.

Reports go out in the background and never hold up the open: a report
that does not fit in the queue is dropped, and drops and failures are
logged once a minute. A filer that answers Unimplemented pauses
reporting for 10 minutes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* mount: address review of the access report

- Drop reports still queued when reporting pauses; the filer that answered
  Unimplemented would answer them the same way.
- Bound each filer attempt to 2s, so a filer that accepts the connection
  but never answers leaves time for the next one, and keep using the filer
  that answered.
- Fail over only when a filer cannot be reached. Any answer, Unimplemented
  included, ends the walk.
- Report an O_CREAT open of an existing file with read access, which comes
  through Create instead of Open. A new file has nothing to read and is not
  reported.
- Stop the reporter on unmount: stop ends the workers and cancels the
  report in flight.
- Say in the flag help that it has no effect unless the filer implements
  TouchAccessTime.
- Tests no longer wait a fixed time to see that nothing else was sent; they
  look at the queue of a reporter that is not started.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* mount: fail an access report over only when the filer did not answer

A status the filer sent ends the walk even when its message contains a
word the transient classifier matches, so an Unimplemented answer still
pauses reporting and a rejected request is not resent elsewhere.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* mount: keep a filer selection made while an access report failed over

A report that reached a later filer moves the mount there only if the
selection it started from is still current, so a slower report cannot
undo another call's failover.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* mount: share an access report's time with the filers not yet tried

Each attempt is capped by an equal share of the report time left, so a
row of stalled filers cannot use up the report before a healthy one later
in the list is tried.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* mount: report the path an open for reading came through

A create that opens an existing hard link reports the name it was
given rather than the inode's first known name. Open reads the inode's
path from the inode table rather than the handle's saved path, which
other handle operations may rewrite concurrently.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* mount: check the access report stop and failover tests without races

The post-stop report is checked on a reporter without workers, so no
worker can take an accepted report before the check, and the failover
test asserts that the stalled filer was tried first.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* pb: keep the shared channel when the peer answers with its own status

gRPC never raises InvalidArgument, NotFound, AlreadyExists,
FailedPrecondition, OutOfRange or DataLoss itself, so such a status is
the peer's answer. It fell through to the string matching, and a message
naming a path like /transport closed the cached channel and cancelled
every other call on it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* mount: say which filer selection an access report keeps

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* mount: bound the in-flight call in the rejected-report channel test

The call held in flight went through touchAccessTime, whose 2 s attempt
cap could expire on a slow runner and fail the test on a healthy channel.
It now uses callFiler with the test deadline. The wait for it to reach
the filer also ends if it returns early or the deadline passes, and the
blocked handler is released on every exit path.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* mount: report each file at most once per -accessReport.interval

Every read open used to send a TouchAccessTime, adding a filer metadata
call per open. A path reported within the interval (default 1h) is now
skipped; the last report times live in a bounded LRU. 0 keeps reporting
every open.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* mount: report an open again once the file was modified since its last report

Within -accessReport.interval a read open was skipped even after a write,
so the filer kept an atime older than the mtime until the interval ran
out. relatime records that first read after a write; do the same by
comparing the entry's mtime with the last report.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* mount: keep the report time apart from the mtime it saw

Recording max(now, mtime) as the last report let a file dated a day ahead
suppress reports for a day plus the interval. Record when the report was
sent and the mtime the entry had, and treat any change of that mtime as a
modification, so neither clock can stretch or shorten the interval.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: MapleWhiteZZZ <18653045+MapleWhiteZZZ@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
2026-11-18 14:11:11 -08:00
Chris LuandClaude Opus 5.5 9f413ff2c8 shell: volume.check.disk skips replicas whose live counts match (#12743)
* shell: volume.check.disk skips replicas whose live counts match

FileCount counts every write and DeleteCount every delete or overwrite
until a vacuum drops them, so a replica vacuumed while its peer was not
keeps a different FileCount forever and was re-checked on every run
although both hold the same live needles. Only treat counts as differing
when the live counts (FileCount - DeleteCount) differ too.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* shell: volume.check.disk skips equal live counts only across compaction revisions

Equal live counts at the same compaction revision mean a missed write
and a missed delete, not a vacuum-state difference, so keep checking
those pairs.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* shell: volume.check.disk real-time recheck ignores heartbeat revisions

VolumeStatus counts are newer than the heartbeat's compaction revisions,
so a vacuum in between could make a pair at the same revision look
vacuum-skewed. Apply the old count rules there.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* shell: volume.check.disk skips on live counts only when no repair could follow

Equal live counts across compaction revisions can still hide a missed
write and a missed delete. Keep checking such pairs when -syncDeleted
could propagate the delete, or when -resurrectMissingNeedles could copy
a needle into a never-vacuumed replica.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-18 14:10:47 -08:00
e037a9c1ce s3vectors: enforce AWS metadata, key and filter limits (#12738)
* s3vectors: enforce AWS metadata, key and filter limits

PutVectors accepted vector metadata of any size and key count, bounded
only by the request body limit, and keys of any length; QueryVectors
accepted filters of any size. Apply the AWS S3 Vectors limits
(https://docs.aws.amazon.com/AmazonS3/latest/userguide/s3-vectors-limitations.html),
taking the ENHANCED index values where the modes differ, since this
implementation prefilters and takes no indexMode:

- 100 KB total metadata per vector, which also bounds a single
  non-filterable field;
- 2 KB filterable metadata per vector;
- 50 metadata keys per vector;
- 1024-character vector keys;
- at most 10 nonFilterableMetadataKeys of 1 to 63 characters at
  CreateIndex;
- 100 filter constraints per QueryVectors request.

A violation is a ValidationException before the engine is called.

* s3vectors: accept metadata whose JSON is exactly at the byte limits

metadataEntrySize counts a trailing comma for every member, so starting
from both braces overcounted by one and rejected metadata that encodes to
exactly 2 KB filterable or 100 KB total.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3vectors: rewrap the metadata limits header comment

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-18 13:49:15 -08:00
Denis 3db54ee7e5 kms/openbao: refresh the AppRole token before it expires (#12734)
* kms/openbao: refresh the AppRole token before it expires

The provider logged in with AppRole once, when it was created, and
never renewed the token. After the token's TTL every SSE-KMS upload,
and every read of an SSE-KMS object whose data key was not cached,
failed with "permission denied" (HTTP 500 to the client) until the S3
server was restarted.

With AppRole authentication the provider now logs in again in the
background once two thirds of the token's TTL have passed, and retries
a failed login every 5 s. Each login is bounded by the client's
timeout, or by 30 s when request_timeout is not positive. Close stops
the refresh. A static token, or an AppRole token without a TTL, is left
as it is.

* kms: keep the OpenBao token refresh across configuration reloads

- AddKMSProvider and InitializeGlobalKMS create a provider instance of
  their own with the new NewProvider instead of the registry's cached
  one: a reload closed the cached instance and then got it back, which
  now stopped its token refresh. The new instance is created before the
  old one is closed, so a configuration that fails leaves the current
  provider working
- openbao: time each refresh from the start of the login that issued
  the token, since its TTL runs from then rather than from the response
- tests: a reload gives a new instance and closes the old one, a failed
  reload keeps the old one; the refresh survives a reload; a slow login
  response

* kms: create a replacement provider outside the manager lock

AddKMSProvider ran the provider factory, which for OpenBao is a login
that may wait for its timeout, while holding the manager lock, so every
lookup, also of unrelated providers, waited for it. The new provider is
now created before the lock is taken; the lock covers only the swap, and
the replaced provider is closed after it. Test: a lookup while another
provider is being created.
2026-11-18 11:07:09 -08:00
Chris LuandClaude Opus 5.5 94de65b3b5 s3api: answer a malformed Cache-Control on PutObject with InvalidRequest (#12733)
PutObject rejected a Cache-Control value it could not parse with
InvalidDigest, whose message is about Content-MD5, even when the request
sent no Content-MD5. CreateMultipartUpload already answers the same parse
error with InvalidRequest; PutObject now does too.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-18 11:03:57 -08:00
ebc77400ca operation: reassign an upload after a proven-expired volume JWT (#12729)
* operation: reassign an upload after a proven-expired volume JWT

* seaweed-volume: mark an expired write JWT refused before append

Mirror the Go volume server: a multipart POST whose write JWT is signed
with the current key, names this fid and fails only on exp gets
X-Seaweed-Write-Outcome: expired-jwt-before-append on its 401, so the
uploader reassigns instead of failing. Both checks use one guard
snapshot, so a key rotation between them cannot drop the marker.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* operation: keep same-FID retry for filer-proxy expiries, snapshot the write signing key, and make the RetryForever budget test fail fast

* operation: re-mint the filer credential before a proxy same-FID expiry retry

* operation: make a proxy marked-expiry terminal in the outer retry too, not only the inner same-FID loop

---------

Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-18 11:02:52 -08:00
Chris LuandClaude Opus 5.5 ecbf7af1ca mount: compare permissions against local ids without mapping them again (#12726)
* mount: compare permissions against local ids without mapping them again

Access, open and create (without -defaultPermissions) ran FilerToLocal on
entries that maybeReadEntry and maybeLoadEntry already return in local ids.
Under a uid/gid map where a local id is also a mapped filer id, the owner
was mapped twice and the check refused the file's own owner.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* mount: cache a deferred create's placeholder in filer ids

The meta cache stores filer uid/gid and maps them to local ids on every
read, but createRegularFile cached the deferred or writeback placeholder
with the caller's local ids. The second mapping in the permission checks
used to cancel that out under a swapping id map; without it the owner of
a placeholder was refused. Cache a mapped clone and keep the handle's
entry in local ids.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* mount: test that a deferred create returns its entry in local ids

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-18 11:01:30 -08:00
b64895ed0a filer: stop following a peer that stays unreachable (#12725)
* filer: stop following a peer that stays unreachable

* filer: re-track an answering peer only while it is still followed

A signature read that succeeds just before the master removes the peer
called initPeerWatermark after the removal, clearing its mark. The loop
then exits, so the watermark it re-tracked never advances or drops and
pins aggregated readers at the settled horizon.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* filer: forget a peer's unreachable clock when the master removes it

Only a successful connection cleared the clock, so every departed address
that once failed stayed in the map for the filer's lifetime, and a re-added
peer inherited the old outage. The removal now drops it, and the clock is
guarded by peerChansLock so the removed peer's exiting loop cannot start a
new one.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* filer: report the unreachable mark only when it is set

Once peerResumeRetention forgets the mark of a peer that stays
unreachable, markPeerWatermarkRemoved no longer marks it (its watermarks
are gone), but notePeerUnreachable still saw no mark and logged
"treating it as removed" on every retry. markPeerWatermarkRemoved now
reports whether it set a new mark.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test: fix the unreachable-peer subscriber's start before the baseline write

The subscriber goroutine picked its start time itself, so a baseline write
that finished first fell before the subscription and the test failed after
20 seconds with correct delivery.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-18 11:00:15 -08:00
72ac9e96b2 mount: answer an interrupted read with EINTR, not EIO (#12723)
* mount: answer an interrupted read with EINTR, not EIO

The kernel's FUSE contract (Documentation/filesystems/fuse/fuse.rst,
"Interrupting filesystem operations") lets a filesystem either ignore an
INTERRUPT or honor it by replying EINTR to the original request. Read did
neither: the FUSE cancel channel cancels the read's context, the chunk
fetch fails with "context canceled", and every error became EIO.

Interrupts come from any signal the reader handles. Go's runtime sends
SIGURG for goroutine preemption, so Go programs reading under load hit
this on healthy data: on a 128-thread 64 KiB random-read test about 250
of 5 million reads failed with EIO, and none with GODEBUG=asyncpreemptoff=1.

Read now returns EINTR when its own context was cancelled, the way
dirReadStatus already does for directory reads and lookups do since
#11717. Go's os.File retries EINTR, so those readers see no error. A
cancelled read also skips the entry refresh retry, and neither read log
line reports an interrupt as a warning or error. Failures without an
interrupt stay EIO.

Tests: TestReadInterruptedReturnsEINTR (fails on the old code with EIO),
TestReadFailureWithoutInterruptReturnsEIO; weed/mount/... passes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* mount: require the failed-read test to reach the filer

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* mount: skip the read retry once the request is interrupted

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
2026-11-18 10:58:05 -08:00
11af0e0c12 mount: move an open file's read view to the chunks a merge wrote (#12722)
* mount: move an open file's read view to the chunks a merge wrote

When a flush finds that a file's chunks take more than twice its size,
maybeMergeChunks reads the file back and writes it again as new chunks.
The commit that publishes them makes the filer delete the chunks they
replaced. The flush put the new chunks into the handle's entry but left
its read view on the old ones, so every descriptor that kept the file
open went on reading the replaced chunks. Those read back until their
volume is vacuumed. After that a read first retries the fetch of the
missing needle for about 13 s; then refreshEntryForRead installs the
filer's entry and the read is retried. The refresh skips a handle with
writes it has not flushed, and such a read returns EIO instead.

Files that are kept open and rewritten in place, such as PostgreSQL
data files and SQLite databases, are the ones whose chunks bloat, so
they are the ones that hit this.

installRewrittenChunks puts the merge's chunks into the entry and the
read view together, under the entry lock. Chunks attached while the
merge ran were not part of it and stay. Flushes without a merge are
unchanged.

TestMergeMovesReadViewOffReplacedChunks covers the handle.
TestReadOpenHandleAfterChunkMerge runs it on a real mount, with and
without an unflushed write; for it, the fuse_integration framework
starts mini on a free master port and exposes the address.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* mount: drain uploads before a merge replaces the chunks

Uploads attach their chunks without the handle lock, first to the entry
and then to the read view, and writes that landed after doFlush drained
the pages may still be uploading when the flush takes its lock. An
upload that attached during the merge could keep a chunk older than
data the merge already holds, or put a chunk the merge replaced back
into the read view. Either outranks the merge's zero-timestamp chunk.

When the chunks are bloated enough to merge, the flush now drains the
pages again while its lock holds writes off, and takes the chunks again.
FlushAll returns only after every upload has attached to both the entry
and the read view, so nothing attaches during the merge.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* mount: leave a pre-merge upload failure to the next flush

The drain before a merge returned an upload failure through the
metadata flush. doFlush then retried it as a filer error and answered
EIO, where the page flush answers a full disk with ENOSPC. The drain is
there only to wait for the uploads: a failure stays with the pages, and
the next flush reports it as a write error, as it would without a merge.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* mount: keep chunks attached while a flush installs its compacted list

A write that lands after doFlush drained the pages can still be uploading
when flushMetadataToFiler takes its chunk snapshot. Its upload attaches the
chunk without the handle lock, and the plain flush then replaced the
entry's chunks with the list compacted from the snapshot, dropping it from
the handle and from every later commit. Keep chunks attached after the
snapshot, as the merge path already does.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* mount: fail a merge's flush on a failed drain

A failed upload found by the drain before a merge was only logged, and
the flush went on to publish a size covering bytes that no chunk stores.
It now fails the flush as a pageUploadError before anything is
published; doFlush maps it with writeErrorToFuseStatus, as it does its
own FlushData failure, and does not retry it as a filer error.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* mount: clone a flush's request entry under the entry lock

The request snapshot cloned the live entry after GetEntry had released
its read lock. Uploads attach chunks without the handle lock, so an
AppendChunks that landed during the clone raced it on the Chunks field.
Clone inside UpdateEntry, as content flush recovery does.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* mount: leave a merge to a later flush while uploads are pending

Before a merge the flush drained the pages while it held the handle and
publication locks. A writeback upload retries a transient storage error
until it succeeds, so that drain could hold both locks for as long as
storage stayed down, stalling reads, writes and stat on the handle.
The flush now checks for pending uploads and, if any, publishes the
compacted chunks and leaves the merge to a later flush. With none
pending the lock still holds writes off, so nothing attaches during the
merge, and a failed upload still fails the flush as a write error.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* mount: keep room for pending uploads when a flush merges onto a moved entry

A flush that leaves the chunk merge to a later flush publishes while
uploads are still pending, and writes that landed after the drain can be
uploading on any flush. Merging a peer's inline content or a shrink into
such a publication lets the upload attach its chunk next to the inline
bytes, where reads never reach it. Tell createContentEntry when uploads
are pending so the merge keeps room for them.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* mount: test chunk merges through the filer's chunk proxy, also under load

The tests serve the filer's chunk proxy, and like the filer the fake
deletes the chunks each commit replaces, so a read view still naming
them fails:
- TestFlushMergeThroughChunkProxy/Quiet: the flush merges, and reads
  through the open handle return the file from the merged chunk alone.
- TestFlushMergeThroughChunkProxy/UploadInFlight: an older page is still
  uploading while a newer page of the same bytes has attached. The flush
  leaves the merge, reads return the newer page, and the next flush
  merges.
- TestFlushMergeRacesWithWritesAndReads: four writers, three flushers
  and a reader on one handle, with uploads delayed at random. The
  writers pause now and then, so 14 to 18 flushes a run merge and the
  others leave the merge to a later one. Every read must succeed within
  2 s, and at the end the handle and the filer must both hold exactly
  what was written.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* mount: let a held upload go when UploadInFlight fails

The subtest's proxy holds the older page's upload until the test
releases it, and only the passing path did. A failure in between left
the handler blocked, so the proxy's cleanup waited on it and the failure
surfaced as a suite timeout. Release it from a cleanup as well, once,
and bound the waits for the upload and the drains.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
2026-11-18 10:51:31 -08:00